Store the depth channel in an analysis result and serve it to the 2D depth view
A replay could draw arrows but not depth, and the recording view said there was nothing to draw even when a result existed. Depth is now quantised per frame (14 B empty, 486 B at the worst frame seen), read from the cache by the same draw path live uses, and the raw-recording note appears only when no result exists. Format version 2; see apps/studio/docs/replay-analysis.md.
This commit is contained in:
ykcheungab
@@ -190,7 +190,7 @@
|
||||
"view_mode_tip": "Image display mode while running",
|
||||
"replay_pipeline": "Recorded video (raw camera image)",
|
||||
"replay_pipeline_tip": "Replay shows the recorded raw camera image with no live arrow overlay; to check the algorithm, use View → Recompute forces from the video",
|
||||
"replay_raw_note": "A recording stores only the raw camera image: there is no flow or depth to draw. Use “Recompute from the video” for the algorithm's output",
|
||||
"replay_raw_note": "A recording stores only the raw camera image: there is no flow or depth to draw, because this recording has no analysis result yet. Opening a recording runs the pass automatically, or press “Recompute from the video”; once a result exists this pane can draw arrows and the depth map",
|
||||
"frame_missing_note": "This frame carries no {fields}: there is nothing for this view to draw",
|
||||
"field_flow": "flow",
|
||||
"field_depth": "depth",
|
||||
@@ -201,6 +201,7 @@
|
||||
"arrow_empty_tip": "The deformation arrow image appears here while running or previewing",
|
||||
"depth_unavailable": "No depth data for this frame\n(not a depth of zero, and no silent fallback to the processed image)",
|
||||
"depth_scale": "Depth scale {low} – {high}",
|
||||
"depth_empty": "This frame's depth is all zeros (0 of {pixels} px non-zero): the ramp's low colour is a real zero response, not missing data",
|
||||
"arrow_scale": "Arrows: displacement ×{factor}",
|
||||
"arrow_scale_tip": "Display magnification of the relative displacement: the arrows are drawn from the algorithm output of the captured image, with the measured displacement multiplied by this factor and capped at 0.8 of the grid spacing (a capped arrow no longer shows its true factor).",
|
||||
"arrow_pipeline": "Captured image → algorithm output",
|
||||
|
||||
@@ -190,7 +190,7 @@
|
||||
"view_mode_tip": "运行时的图像显示模式",
|
||||
"replay_pipeline": "录制视频（原始相机图像）",
|
||||
"replay_pipeline_tip": "回放显示的是录制下来的原始相机图像，不含实时箭头叠加；要复核算法请用「视图 → 由视频重算力值」",
|
||||
"replay_raw_note": "录像只保存原始相机图像：这里没有光流/深度可画。要看算法处理结果，请用「由视频重算」",
|
||||
"replay_raw_note": "录像只保存原始相机图像：这里没有光流/深度可画，因为这个录像还没有分析结果。打开录像会自动跑一遍分析，也可以点「由视频重算」手动生成；有结果后本面板就能画箭头和深度图",
|
||||
"frame_missing_note": "本帧没有{fields}：这个视图这一帧没有数据可画",
|
||||
"field_flow": "光流",
|
||||
"field_depth": "深度",
|
||||
@@ -201,6 +201,7 @@
|
||||
"arrow_empty_tip": "运行或预览后在此显示形变箭头图",
|
||||
"depth_unavailable": "本帧没有深度数据\n（不是深度为零，也未回退显示普通形变图）",
|
||||
"depth_scale": "深度色标 {low} – {high}",
|
||||
"depth_empty": "本帧深度全为 0（{pixels} px 中 0 个非零）：色标低端色是一帧真实的零响应，不是缺数据",
|
||||
"arrow_scale": "箭头按位移放大 ×{factor}",
|
||||
"arrow_scale_tip": "相对位移的显示放大倍率：箭头由原始采集图像经算法输出，长度按测量位移放大后绘制，上限为网格间距的 0.8 倍；触顶的箭头不再呈现真实倍率。",
|
||||
"arrow_pipeline": "原始图像 → 算法输出",
|
||||
|
||||
@@ -12,8 +12,12 @@
|
||||
|
||||
| 模式 | 画面来源 | 说明 |
|
||||
|---|---|---|
|
||||
| `recording` | 录像自己的 `raw_*.mp4`，按需解码 | 只有原始相机图像：没有光流，也没有深度 |
|
||||
| `analysis` | 存下来的分析结果，按帧读 | 有 pass 记录下来的光流样本与候选网格 |
|
||||
| `recording` | 录像自己的 `raw_*.mp4`，按需解码 | 这个画面是原始相机图像：它自己没有光流也没有深度 |
|
||||
| `analysis` | 存下来的分析结果，按帧读 | 有 pass 记录下来的光流样本、候选网格与深度图 |
|
||||
|
||||
模式说的是**画面从哪里读**，而"有没有可画的东西"是另一件事：只要有结果，二维面板的模式
|
||||
选择器就提供 深度图/箭头（时间轴在录像上时连"原始图像"一起提供），选了派生模式的那些帧
|
||||
一律从结果读、按窗口唯一的 2D 出口画 —— 绝不会把原始图像当成箭头图交出去。
|
||||
|
||||
图表、二维画布、三维面板都读这一个游标，模式由 `ReplayEngine.mode()` 回答；「从视频重算」
|
||||
因此不再是第三种来源，而是**显式重跑**这一遍的开关（第 10 节）。时间基准是**信号时间**
|
||||
@@ -47,9 +51,16 @@
|
||||
| 视图 | 读什么 | 怎么读 |
|
||||
|---|---|---|
|
||||
| 图表 | `forces.csv`（记录值） | `charts.set_cursor(t)` 用同一游标；结果里不存力 |
|
||||
| 二维画布 | 结果的 `flow` 与 `valid_mask` | `ImageFrame(source="analysis")` 走窗口唯一的 2D 出口 `_show_image_frame`，箭头、暗点、配色与域边界与实时/重算**同一套代码** |
|
||||
| 二维画布 | 结果的 `flow`、`depth` 与 `valid_mask` | `ImageFrame(source="analysis")` 走窗口唯一的 2D 出口 `_show_image_frame`，箭头、暗点、配色、域边界与深度视图与实时/重算**同一套代码** |
|
||||
| 三维面板 | 同一帧的 `SdkPacket` | `session.packet(frame)` → `Scene3DView.submit_packet`，走同一个 adapter、域掩码 latch 与有效性求交 |
|
||||
|
||||
**深度视图也由结果喂。** `AnalysedFrame.depth` 就是第 1 节那幅量化平面重建出来的
|
||||
`float32` 图（空帧是形状正确的全零），`depth_valid` 因此是 `True` 而不像以前那样恒为
|
||||
`False`；深度着色、色标锁定、空/缺数据的区分全在窗口既有的那条路径上，回放没有第二条绘制
|
||||
代码。"这一帧深度是精确的零"由面板说出来（`panels.depth_empty`，带 `{pixels}`），与
|
||||
"这一帧根本没有深度通道"（`panels.depth_unavailable`）是两句不同的话，因为它们是两个不同的
|
||||
事实。
|
||||
|
||||
**画面是重建的，并且是精确重建。** 二维需要的 `[H,W,2]` 流场是把存下来的 `dx`/`dy` 写回
|
||||
**记录下来的候选位置**、其余格子填 `NaN`（选区器把非有限值当作非法，邻居格子因此不可能贡献
|
||||
出 pass 没记录过的样本）。写回要除以
|
||||
@@ -102,6 +113,22 @@ on_progress)`（`chain_spec_for` 从录像自己的 `session.json` 取链路，
|
||||
`test_the_replay_reset_is_the_display_reset_and_never_the_device` 失败过 —— 合成录像的配置
|
||||
缺失让 pass 在断言之前结束并覆盖了 `复位显示` 的提示；修的是消息落在哪里。
|
||||
|
||||
**深度那一片加的验收**（同一个文件再 +8 项，另加 `shared/tests/test_analysis_depth.py` 10 项）：
|
||||
|
||||
* **深度往返**：另跑一遍链路，逐帧比 `depth_map`，误差上界是该帧自己的 `(max − min) / 510`，
|
||||
在 box 录像（33 帧，1 帧非空、76 px）和 264 帧的 `long_8x.mp4`（241 帧空、23 帧非空、
|
||||
最坏 336 px）上都是 0 失败；非零计数、min/max 与链路的完全相等。
|
||||
* **跳转等于顺序播放（深度模式）**：把上面那条测试扩到深度 —— 顺序播放记下每帧渲染出的深度
|
||||
图，再跳到 5 个位置，帧 id、`depth`、统计与**渲染出的字节**逐项相等。
|
||||
* **尺寸**：空帧 4,048 B（固定 4,042 + 6 B 载荷头），最坏非空帧 4,534 B（+486 B）；
|
||||
按实测比例 4,090 B/帧，十分钟 70.2 MB（上限 81.6 MB）。断言写死了这三个数。
|
||||
* **模式切换不写结果**：四个模式各画一次，`scan_results(cache)` 的目录集合不变。
|
||||
* **"没什么可画"只在该说的时候说**：无结果时 `plan_image` 只给 `recorded` 并带
|
||||
`replay_raw_note`；有结果时该行消失、四个模式都在，默认仍是原始图像。两种语言各断言一次，
|
||||
且文案必须点出产生结果的动作。
|
||||
* **派生模式走结果**：时间轴在录像上、模式选深度时，送到 2D 出口的是**结果那一帧**（含深度），
|
||||
不是解码出来的视频帧；切回"原始图像"仍是录像自己的画面。
|
||||
|
||||
真实 GL 快照（`QT_QPA_PLATFORM=windows` + `QT_OPENGL=desktop`，真实窗口路径、pass 写入
|
||||
scratch cache、游标停在存下来的帧 id 上）：
|
||||
|
||||
@@ -115,11 +142,27 @@ scratch cache、游标停在存下来的帧 id 上）：
|
||||
二维画面是候选网格上的暗点阵（该帧活动样本很少），三维是手指岛面加同一套样本点；"同一个样本
|
||||
集"由上面的逐元素断言保证，快照证明这条链路真的走到了 GL。
|
||||
|
||||
**深度那一片的快照**（`.scratch/replay-analysis-a2/snapshots/`，同一个窗口、同一个游标、
|
||||
同一个模式选择器；时间轴停在录像上，所以"原始图像"是真的相机图，两个派生模式读结果）：
|
||||
|
||||
| 文件 | 内容 |
|
||||
|---|---|
|
||||
| `analysis-finger-frame-33-2d-raw.png` / `-arrows.png` / `-depth.png` | finger 链路帧 33，深度非零 76 px（峰值 0.0242） |
|
||||
| `analysis-box-frame-27-2d-raw.png` / `-arrows.png` / `-depth.png` | box 链路帧 27，`depth_map` 365×365 **恰好全零**，面板显示"本帧深度全为 0（133225 px 中 0 个非零）" |
|
||||
|
||||
同一帧三张图的大小不同（raw 96.6 kB 是相机图，arrows 4.9 kB，depth 4.2 kB），这本身就是
|
||||
"三个模式画的是三样东西"的证据。
|
||||
|
||||
## 15. 已知限制
|
||||
|
||||
* 结果里没有力，也没有深度：力曲线始终读记录的 `forces.csv`，深度视图在回放模式下不可用（没有
|
||||
可显示的数据，面板按空处理而不是画一张假的）。
|
||||
* 结果里没有力：力曲线始终读记录的 `forces.csv`。深度有了（第 1、11 节），但**三维面板仍只
|
||||
拿光流**：`session.packet()` 的 `depth` 依旧是 `None`、`depth_valid` 依旧是 `False`，
|
||||
因为把深度接进三维会改变三维图形的构成，而那不在这一片里。二维深度视图是这一片交付的东西。
|
||||
* 结果覆盖的帧是 pass 走过的那些：录像比结果长时游标可以离开结果范围（面板说"这个分析结果里
|
||||
没有可显示的帧"），不会静默重复最后一帧。
|
||||
* 栅格类显示设置不生效（第 12 节）；换栅格就是重跑一遍，那是**另一个键**。
|
||||
* 一个进程同时只有一遍分析在跑；采集会话运行期间不启动分析。
|
||||
* 时间轴与模式选择是两件事，而"有没有结果"是录像的属性、不是游标的属性：
|
||||
`ReplayController.available_result()` 查的是**本录像**在 `ResultLibrary` 里的那一份，所以
|
||||
即使 `ReplayEngine.use_recording()` 让引擎不再借用 session（目前 UI 没有第二条路会走到
|
||||
这个状态），二维面板照样能画箭头和深度，`replay_raw_note` 也不会重新出现。
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
回放过去等于每移动一次光标就重跑一遍 SDK 链路：**每个视频秒 0.48–0.51 秒墙钟**，十分钟录像
|
||||
一次约 4.8 分钟。Part A 把这一遍跑一次，只存回放真正要画的东西。
|
||||
|
||||
格式版本从 1 升到 2：帧记录里多了深度通道。旧结果不会被误读 —— 版本不符直接
|
||||
`AnalysisFormatError`，重跑一遍即可（结果本来就是可重算的）。
|
||||
|
||||
## 1. 结果模型
|
||||
|
||||
原始输出每帧 **1,176,000 B**（`flow` 672,000 + `depth_map` 336,000 + `valid_mask` 84,000 +
|
||||
@@ -15,19 +18,34 @@
|
||||
`select_flow_samples` 选出的样本。
|
||||
|
||||
**每帧记录**（`frame_id`、`t_rel_s`、网格 keyframe 编号、候选数、深度统计、`dx`、`dy`、
|
||||
`magnitude`、位打包 `active`、掩码摘要）在 shipped finger 网格下是 **4,034 B/帧**：
|
||||
`24 + 330×12 + ⌈330/8⌉ + 8`；十分钟 30 fps 约 70 MB。选这个编码而不是实测的另外两种
|
||||
（float16 的 2,022 B/帧、丢掉 `magnitude` 的 4,002 B/帧），是因为往返必须**逐位相等**，而
|
||||
`magnitude` 正是"同一份结果换个阈值也能用"的依据（第 3 节）。`dx`/`dy`/`magnitude` 存
|
||||
`magnitude`、位打包 `active`、掩码摘要、深度载荷）在 shipped finger 网格下是 **4,048 B/帧**
|
||||
（空深度）：`32 + 330×12 + ⌈330/8⌉ + 8 + 6`；十分钟 30 fps 约 70 MB。选这个编码而不是实测的
|
||||
另外两种（float16 的 2,022 B/帧、丢掉 `magnitude` 的 4,002 B/帧），是因为往返必须**逐位相等**，
|
||||
而 `magnitude` 正是"同一份结果换个阈值也能用"的依据（第 3 节）。`dx`/`dy`/`magnitude` 存
|
||||
float32，与实时选择器比对时后者也要转成 float32 才是精确比较。
|
||||
|
||||
**不存的**：力（`forces.csv` 已是图表读的内容，180.9 B/帧）；深度图只存每帧一个
|
||||
`depth_nonzero` 计数（4 B）加汇总统计 —— 这段素材上深度几乎全零（264 帧中 241 帧全零，最大
|
||||
336 px / 84,000），统计记下来"深度视图是空的"才有解释。
|
||||
**深度存下来，而且是量化后存的。** 单帧 `depth_map` 336,000 B，十分钟 6.0 GB —— 直接存不可行；
|
||||
实测这段素材上它绝大多数帧**恰好全零**（264 帧的八倍速片段里 241 帧，box 录像 33 帧里 32 帧），
|
||||
最坏的非空帧也只有 84,000 px 中的 336 px。
|
||||
|
||||
所以每帧固定记下**深度通道的统计**（是否有通道、
|
||||
非零计数、最小/最大值，共 8 B）并附一个深度载荷：空帧只有一个 6 B 的载荷头（声明形状，
|
||||
指明"这一帧的深度是精确的零"），非空帧是**按该帧自己的 [min, max] 量化的 uint8 平面**再
|
||||
zlib level 6。
|
||||
|
||||
实测代价：空帧 +14 B，最坏非空帧 486 B；能在往返中引入的最大误差是半个量化步长
|
||||
`(max − min) / 510`，实测 6.6e-5（该帧峰值 0.034）。这不到深度视图所用 256 级色标的一步，所以
|
||||
重建出来的帧**不可能被染成与原帧不同的颜色**。
|
||||
|
||||
稀疏存储实测过并否决：每像素 4 B 索引在最坏帧上要 910 B，而整幅量化平面只要 486 B —— 一片几乎
|
||||
全零的字节平面在 zlib 下几乎免费，像素索引不是。限制在岛内也实测过并否决：418 B 对 486 B，
|
||||
在 4,048 B 的记录里省 5.2 B/帧，代价是重建要依赖有效性掩码 keyframe（box 链路根本没有），
|
||||
而且一旦有深度值落在岛外就会**静默丢掉一个测量**。
|
||||
|
||||
`provenance` 随结果存：算法版本、传感器配置及其内容哈希、标定 id、单位、参考帧 id、首个输出
|
||||
帧 id、解码偏移、逐通道的"重算/记录"标记（`flow`、`depth` 重算，`forces` 记录）、`complete`
|
||||
标志与计数。
|
||||
标志与计数。`counts` 另记 `depth_frames` / `depth_nonzero_frames` / `depth_nonzero_max` /
|
||||
`depth_pixels` 与实测的 `frame_record_bytes`，进度面板和"这一帧为什么是空的"都读它。
|
||||
|
||||
## 2. Keyframe：候选位置与有效掩码
|
||||
|
||||
|
||||
@@ -2610,11 +2610,26 @@ class StudioMainWindow(QMainWindow):
|
||||
self._depth_range_value = (lo, hi)
|
||||
return self._depth_range_value
|
||||
|
||||
def _depth_scale_text(self) -> str:
|
||||
def _depth_scale_text(self, frame: ImageFrame | None = None) -> str:
|
||||
"""The colour scale, plus — when the frame's depth is exactly zero — why it is flat.
|
||||
|
||||
A depth view of an all-zero map is a flat island in the ramp's low colour, which
|
||||
looks the same as a broken view. The frame knows which it is: the pass stored the
|
||||
channel, its non-zero count and its range, so "0 of 84,000 pixels" can be said
|
||||
instead of leaving the operator to wonder. It is not the same statement as
|
||||
``panels.depth_unavailable``, which is for a frame that carries no depth at all.
|
||||
"""
|
||||
span = self._depth_range()
|
||||
if span is None:
|
||||
return ""
|
||||
return t("panels.depth_scale", low=f"{span[0]:.3f}", high=f"{span[1]:.3f}")
|
||||
text = "" if span is None else t(
|
||||
"panels.depth_scale", low=f"{span[0]:.3f}", high=f"{span[1]:.3f}"
|
||||
)
|
||||
if frame is None or frame.depth is None:
|
||||
return text
|
||||
array = np.asarray(frame.depth)
|
||||
if array.size and not bool(np.any(array)):
|
||||
empty = t("panels.depth_empty", pixels=int(array.size))
|
||||
return f"{text} · {empty}" if text else empty
|
||||
return text
|
||||
|
||||
def _latched_domain_mask(self, frame: ImageFrame):
|
||||
"""The 2D domain boundary to draw, latched for the session.
|
||||
@@ -2694,7 +2709,7 @@ class StudioMainWindow(QMainWindow):
|
||||
self.arrow_panel.set_valid_area(frame.valid_mask is not None)
|
||||
if mode == DEPTH:
|
||||
self._last_depth_frame = frame.depth
|
||||
self.arrow_panel.set_depth_scale(self._depth_scale_text())
|
||||
self.arrow_panel.set_depth_scale(self._depth_scale_text(frame))
|
||||
else:
|
||||
self.arrow_panel.set_depth_scale(None)
|
||||
rendered = render_image(
|
||||
|
||||
@@ -110,6 +110,26 @@ class AnalysedFrame:
|
||||
base_bgr: np.ndarray
|
||||
depth_nonzero: int = 0
|
||||
dropped: bool = False
|
||||
#: The stored depth channel of this frame, or ``None`` when the pass recorded none.
|
||||
#: An all-zero map is a *measurement* — "the response was zero everywhere" — and is
|
||||
#: handed over as an array; the depth view draws it and the statistics that came
|
||||
#: with it say why it is flat. See ``analysis.model`` for the encoding.
|
||||
depth: np.ndarray | None = None
|
||||
depth_min: float = 0.0
|
||||
depth_max: float = 0.0
|
||||
|
||||
@property
|
||||
def has_depth(self) -> bool:
|
||||
return self.depth is not None
|
||||
|
||||
@property
|
||||
def depth_pixels(self) -> int:
|
||||
return 0 if self.depth is None else int(self.depth.size)
|
||||
|
||||
@property
|
||||
def depth_is_empty(self) -> bool:
|
||||
"""True when this frame has a depth channel and it is exactly zero everywhere."""
|
||||
return self.depth is not None and self.depth_nonzero == 0
|
||||
|
||||
@property
|
||||
def domain_hw(self) -> tuple[int, int]:
|
||||
@@ -342,6 +362,7 @@ class AnalysisSession:
|
||||
samples = payload.samples(threshold=self.threshold())
|
||||
flow = self._recover_flow(samples, shape_hw)
|
||||
base = np.zeros((shape_hw[0], shape_hw[1], 3), dtype=np.uint8)
|
||||
depth = getattr(payload, "depth", None)
|
||||
return AnalysedFrame(
|
||||
frame_id=int(payload.frame_id),
|
||||
t_rel_s=float(payload.t_rel_s),
|
||||
@@ -351,6 +372,9 @@ class AnalysisSession:
|
||||
base_bgr=base,
|
||||
depth_nonzero=int(getattr(payload, "depth_nonzero", 0) or 0),
|
||||
dropped=bool(getattr(payload, "dropped", False)),
|
||||
depth=None if depth is None else np.asarray(depth, dtype=np.float32),
|
||||
depth_min=float(getattr(payload, "depth_min", 0.0) or 0.0),
|
||||
depth_max=float(getattr(payload, "depth_max", 0.0) or 0.0),
|
||||
)
|
||||
|
||||
def _recover_flow(
|
||||
|
||||
@@ -28,7 +28,12 @@ from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
import numpy as np
|
||||
from orisys_app_shared.acquisition.image_backend import SOURCE_ANALYSIS, SOURCE_REPLAY, ImageFrame
|
||||
from orisys_app_shared.acquisition.image_backend import (
|
||||
DERIVED_MODES,
|
||||
SOURCE_ANALYSIS,
|
||||
SOURCE_REPLAY,
|
||||
ImageFrame,
|
||||
)
|
||||
from orisys_app_shared.i18n import t
|
||||
from orisys_app_shared.paths import recordings_dir
|
||||
from PySide6.QtCore import QObject, QTimer
|
||||
@@ -891,17 +896,75 @@ class ReplayController(QObject):
|
||||
self._actions.recompute.blockSignals(False)
|
||||
self._toolbar.set_recompute_checked(bool(value))
|
||||
|
||||
def available_result(self):
|
||||
"""The stored result for the recording that is open, if there is one.
|
||||
|
||||
Deliberately not ``engine.analysis()``: that answers "which session is the
|
||||
*timeline* borrowing", and it is cleared when the timeline goes back to reading
|
||||
the video (``ReplayEngine.use_recording``). A result is a property of the
|
||||
recording, not of the cursor's mode, and it is the question the panel's one line
|
||||
has to be answered from — "is there a result?" is why the arrows and the depth
|
||||
map can be drawn at all. The session the library holds is checked against the
|
||||
recording that is open, so a previous recording's result cannot answer for this
|
||||
one.
|
||||
"""
|
||||
session = self._engine.analysis()
|
||||
if session is not None:
|
||||
return session
|
||||
held = self.results.session
|
||||
if held is None:
|
||||
return None
|
||||
current = self.recording_directory()
|
||||
stored = getattr(held, "recording", None)
|
||||
if current is None or stored is None or Path(stored) != Path(current):
|
||||
return None
|
||||
return held
|
||||
|
||||
# ------------------------------------------------------------------ drawing
|
||||
def _analysis_image_frame(self, frame, *, source: str) -> ImageFrame:
|
||||
"""One stored frame in the shape the window's single 2D sink takes.
|
||||
|
||||
The *only* place a replay builds a 2D frame from a result, so the analysis
|
||||
timeline and a recording whose mode selector asks for a derived view cannot
|
||||
build two slightly different pictures. ``source`` says which timeline the
|
||||
operator is on — ``replay`` keeps the raw view on offer while the derived modes
|
||||
are drawn from the result, so choosing a derived view is not a one-way door.
|
||||
"""
|
||||
return ImageFrame(
|
||||
frame.base_bgr,
|
||||
flow=frame.flow,
|
||||
depth=frame.depth,
|
||||
valid_mask=frame.valid_mask,
|
||||
source=source,
|
||||
frame_id=int(frame.frame_id),
|
||||
t_rel_s=float(frame.t_rel_s),
|
||||
flow_valid=True,
|
||||
flow_frame_id=int(frame.frame_id),
|
||||
depth_valid=bool(frame.has_depth),
|
||||
depth_frame_id=int(frame.frame_id) if frame.has_depth else None,
|
||||
# The result this frame came from is the thing that makes arrows and depth
|
||||
# drawable at all; the plan reads it to decide what to offer.
|
||||
result_available=True,
|
||||
)
|
||||
|
||||
def render_frame(self) -> None:
|
||||
"""Draw what the 2D panel should show for the current replay cursor.
|
||||
|
||||
One method for both timeline modes, because it is one cursor: in analysis mode
|
||||
the picture is the stored frame's own flow field (the pass's candidate
|
||||
locations, on the domain grid), and in recording mode it is the decoded video
|
||||
frame, which carries no flow and no depth — so the plan says so in one line and
|
||||
the panel offers exactly the one view the frame can be. The window's single 2D
|
||||
sink draws both through the *same* backend, so the arrows, the idle dots, the
|
||||
colour range and the domain boundary behave identically in the two modes.
|
||||
the picture is the stored frame's own flow field and depth map (the pass's
|
||||
candidate locations, on the domain grid), and in recording mode it is the
|
||||
decoded video frame. The window's single 2D sink draws both through the *same*
|
||||
backend, so the arrows, the idle dots, the colour range and the domain boundary
|
||||
behave identically in the two modes.
|
||||
|
||||
**A mode that needs derived data is served from the result, never faked.** The
|
||||
arrows and the depth map are the pass's measurements; with a result in hand they
|
||||
are read from it and drawn through this same path, from the cursor's own frame.
|
||||
The recording's own picture answers only for the mode that *is* the recording's
|
||||
picture, so no mode can silently present the raw image as an arrow plot. With no
|
||||
result there is nothing to serve: the plan offers the recorded view alone, and
|
||||
the panel says so — naming the action that produces a result — instead of leaving
|
||||
the operator to guess why the arrow view is empty.
|
||||
|
||||
Nothing here writes anything: a display change reaches this method through the
|
||||
engine's ``redraw()`` and rebuilds the picture from the same stored frame.
|
||||
@@ -911,30 +974,34 @@ class ReplayController(QObject):
|
||||
if self._recompute_running:
|
||||
# The pipeline is drawing its own frames (see ``owns_image``).
|
||||
return
|
||||
if self.analysis_active():
|
||||
frame = self._engine.analysed_frame()
|
||||
if frame is None:
|
||||
self._image_panel.clear_preview(self._engine.video_message())
|
||||
return
|
||||
self._host._show_image_frame(
|
||||
ImageFrame(
|
||||
frame.base_bgr,
|
||||
flow=frame.flow,
|
||||
valid_mask=frame.valid_mask,
|
||||
source=SOURCE_ANALYSIS,
|
||||
frame_id=int(frame.frame_id),
|
||||
t_rel_s=float(frame.t_rel_s),
|
||||
flow_valid=True,
|
||||
flow_frame_id=int(frame.frame_id),
|
||||
depth_valid=False,
|
||||
session = self.available_result()
|
||||
wanted = str(getattr(self._image_panel, "view_mode", lambda: "")() or "")
|
||||
if session is not None:
|
||||
if self.analysis_active():
|
||||
frame = self._engine.analysed_frame()
|
||||
if frame is None:
|
||||
self._image_panel.clear_preview(self._engine.video_message())
|
||||
return
|
||||
elif wanted in DERIVED_MODES:
|
||||
# The timeline reads the video, but this view is not the video: it is a
|
||||
# measurement, and the result is where measurements live.
|
||||
frame = session.frame_at(self._engine.position())
|
||||
if frame is None:
|
||||
self._image_panel.clear_preview(self._engine.video_message())
|
||||
return
|
||||
else:
|
||||
frame = None
|
||||
if frame is not None:
|
||||
self._host._show_image_frame(
|
||||
self._analysis_image_frame(
|
||||
frame,
|
||||
source=SOURCE_ANALYSIS if self.analysis_active() else SOURCE_REPLAY,
|
||||
)
|
||||
)
|
||||
)
|
||||
session = self._engine.analysis()
|
||||
if session is not None:
|
||||
# The same frame, in the shape the 3D pane's producer consumes: one
|
||||
# cursor, two figures, one frame id.
|
||||
self._host._show_analysis_frame(session.packet(frame))
|
||||
return
|
||||
return
|
||||
image = self._engine.video_frame_bgr()
|
||||
if image is None:
|
||||
self._image_panel.clear_preview(self._engine.video_message())
|
||||
@@ -944,6 +1011,10 @@ class ReplayController(QObject):
|
||||
image,
|
||||
source=SOURCE_REPLAY,
|
||||
t_rel_s=float(self._engine.position()),
|
||||
# A result exists, so the derived views are offered and are drawn from
|
||||
# it (above). Without one there is nothing behind the picture and the
|
||||
# panel says exactly that.
|
||||
result_available=session is not None,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -28,6 +28,19 @@ from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from orisys_app_shared.acquisition.arrows import depth_map_to_bgr, flow_colormap
|
||||
from orisys_app_shared.acquisition.image_backend import (
|
||||
ARROW_PURE,
|
||||
ARROW_STITCHED,
|
||||
DEPTH,
|
||||
DERIVED_MODES,
|
||||
RAW_NOTE_KEY,
|
||||
SOURCE_ANALYSIS,
|
||||
SOURCE_REPLAY,
|
||||
ImageFrame,
|
||||
plan_image,
|
||||
render_image,
|
||||
)
|
||||
from orisys_app_shared.analysis.cache import ResultCache, scan_results
|
||||
from orisys_app_shared.analysis.reader import (
|
||||
AnalysisIncompleteError,
|
||||
@@ -254,10 +267,22 @@ class _ImagePanel:
|
||||
def __init__(self) -> None:
|
||||
self.kinds: list[str] = []
|
||||
self.cleared: list[str] = []
|
||||
#: The 2D mode selector's current value. ``recorded`` is what the panel starts
|
||||
#: on for a recording, so a stub that never sets it behaves as before.
|
||||
self._mode = "recorded"
|
||||
|
||||
def set_source_kind(self, kind: str) -> None:
|
||||
self.kinds.append(str(kind))
|
||||
|
||||
def view_mode(self) -> str:
|
||||
return self._mode
|
||||
|
||||
def set_view_mode(self, mode: str) -> None:
|
||||
self._mode = str(mode)
|
||||
|
||||
def set_plan(self, plan) -> None:
|
||||
self.plan = plan
|
||||
|
||||
def clear_preview(self, message: str) -> None:
|
||||
self.cleared.append(str(message))
|
||||
|
||||
@@ -904,3 +929,266 @@ def test_the_worker_reports_progress_and_a_cancelled_outcome():
|
||||
assert report.cancelled is True and report.complete is False
|
||||
assert report.frames_written == 2
|
||||
assert report_from(None, cancelled=True).cancelled is True
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ the depth view
|
||||
def _analysis_frame_for(frame, *, source=SOURCE_ANALYSIS):
|
||||
"""The frame the controller hands the window's single 2D sink for a stored frame."""
|
||||
return ImageFrame(
|
||||
frame.base_bgr,
|
||||
flow=frame.flow,
|
||||
depth=frame.depth,
|
||||
valid_mask=frame.valid_mask,
|
||||
source=source,
|
||||
frame_id=int(frame.frame_id),
|
||||
t_rel_s=float(frame.t_rel_s),
|
||||
flow_valid=True,
|
||||
flow_frame_id=int(frame.frame_id),
|
||||
depth_valid=bool(frame.has_depth),
|
||||
depth_frame_id=int(frame.frame_id) if frame.has_depth else None,
|
||||
result_available=True,
|
||||
)
|
||||
|
||||
|
||||
def _drawn(frame, mode: str):
|
||||
image = _analysis_frame_for(frame)
|
||||
return render_image(
|
||||
image, plan_image(image), mode=mode, theme="dark", draw_style="box"
|
||||
)
|
||||
|
||||
|
||||
def test_the_depth_view_is_drawn_from_the_cache_by_the_backend_live_uses(session):
|
||||
"""The depth mode renders the stored map through the same call live renders through.
|
||||
|
||||
Nothing here is a second draw path: ``render_image`` in ``depth`` mode is the same
|
||||
function a live frame reaches, over the map the reader handed back. The check that it
|
||||
is really that map is a direct comparison with the shared converter, and the check
|
||||
that it is not some other view is that the arrows picture differs from it.
|
||||
"""
|
||||
frame = session.frame(FIRST_FRAME_ID + 1)
|
||||
assert frame.has_depth, "the pass recorded no depth channel for this frame"
|
||||
assert frame.depth.shape == (300, 280)
|
||||
assert frame.depth.dtype == np.float32
|
||||
|
||||
drawn = _drawn(frame, DEPTH)
|
||||
assert drawn is not None
|
||||
expected = depth_map_to_bgr(
|
||||
frame.depth, flow_colormap("dark"), value_range=None, invalid_mask=frame.valid_mask
|
||||
)
|
||||
assert np.array_equal(drawn, expected), "the depth mode did not draw the stored map"
|
||||
arrows = _drawn(frame, ARROW_PURE)
|
||||
assert arrows is not None
|
||||
assert not np.array_equal(drawn, arrows), "the depth view drew the arrow view"
|
||||
|
||||
|
||||
def test_a_seek_equals_playing_forward_in_the_depth_view(session):
|
||||
"""The seek test, in the depth mode: same frame, same map, same bytes on screen.
|
||||
|
||||
Playing forward records each frame's rendered depth picture; a fresh session is then
|
||||
seeked to five stored timestamps and must produce those pictures byte for byte. The
|
||||
tolerance is zero on the frame id, on ``depth``, on the statistics that travel with
|
||||
it and on the rendered image — because a seek is a file read of the same record.
|
||||
"""
|
||||
ids = [int(value) for value in session.frame_ids]
|
||||
played: dict[int, tuple] = {}
|
||||
for frame_id in ids:
|
||||
frame = session.frame(frame_id)
|
||||
played[frame_id] = (
|
||||
frame.depth,
|
||||
frame.depth_nonzero,
|
||||
frame.depth_min,
|
||||
frame.depth_max,
|
||||
_drawn(frame, DEPTH),
|
||||
)
|
||||
fresh = AnalysisSession(
|
||||
AnalysisReader(_result_directory(Path(session.directory).parent)),
|
||||
policy=session.policy,
|
||||
display_policy=session.display_policy,
|
||||
)
|
||||
try:
|
||||
nonempty = 0
|
||||
for frame_id in (ids[0], ids[1], ids[len(ids) // 2], ids[-2], ids[-1]):
|
||||
moment = fresh.time_of(frame_id)
|
||||
assert moment is not None
|
||||
seeked = fresh.frame_at(moment)
|
||||
assert seeked is not None and seeked.frame_id == frame_id
|
||||
depth, nonzero, low, high, picture = played[frame_id]
|
||||
nonempty += 1 if nonzero else 0
|
||||
assert seeked.depth_nonzero == nonzero
|
||||
assert (seeked.depth_min, seeked.depth_max) == (low, high)
|
||||
assert np.array_equal(np.asarray(seeked.depth), np.asarray(depth))
|
||||
picture_now = _drawn(seeked, DEPTH)
|
||||
if picture is None or picture_now is None:
|
||||
assert picture is None and picture_now is None
|
||||
else:
|
||||
assert np.array_equal(picture_now, picture)
|
||||
assert nonempty <= 5
|
||||
finally:
|
||||
fresh.reader.close()
|
||||
|
||||
|
||||
def test_a_mode_change_writes_no_analysis(pass_root, session):
|
||||
"""Switching the 2D mode is a read: the result directory set cannot move.
|
||||
|
||||
The depth map is in the result, so drawing it is a file read like the arrows are.
|
||||
The assertion is the set of result directories before and after every mode the
|
||||
selector offers is selected, drawn and switched away from.
|
||||
"""
|
||||
before = {entry.directory.name for entry in scan_results(pass_root)}
|
||||
for mode in (DEPTH, ARROW_PURE, ARROW_STITCHED, "recorded"):
|
||||
frame = session.frame(FIRST_FRAME_ID + 1)
|
||||
if mode != "recorded":
|
||||
assert _drawn(frame, mode) is not None
|
||||
assert {entry.directory.name for entry in scan_results(pass_root)} == before
|
||||
|
||||
|
||||
def test_the_depth_statistics_say_which_frames_are_empty(session):
|
||||
"""An empty depth view is explainable: the frame says so, and so does the result."""
|
||||
frames = list(session.frames())
|
||||
emptied = [item for item in frames if item.depth_is_empty]
|
||||
carrying = [item for item in frames if item.depth_nonzero]
|
||||
assert len(emptied) + len(carrying) == len(frames)
|
||||
assert all(item.has_depth for item in frames), "a frame lost its depth channel"
|
||||
assert all(item.depth_pixels == 300 * 280 for item in frames)
|
||||
# Every observed frame on this footage falls in one of the two groups, and the empty
|
||||
# ones are exactly zeros rather than NaNs or a shape that failed to decode.
|
||||
for item in emptied:
|
||||
assert not bool(np.any(np.asarray(item.depth)))
|
||||
assert (item.depth_min, item.depth_max) == (0.0, 0.0)
|
||||
for item in carrying:
|
||||
assert float(np.asarray(item.depth).max()) == item.depth_max > 0.0
|
||||
|
||||
|
||||
# ------------------------------------------------------- the operator's one line
|
||||
def _note_for(image: ImageFrame) -> str:
|
||||
plan = plan_image(image)
|
||||
return "" if not plan.note_key else t(plan.note_key, fields="flow、depth")
|
||||
|
||||
|
||||
def test_the_nothing_to_draw_line_appears_only_when_no_result_exists():
|
||||
"""The recording's note is an absence statement, and it must be true when it is made.
|
||||
|
||||
Two recording frames that differ in exactly one fact: whether a stored analysis
|
||||
result exists. Without one, the pane really can draw nothing derived, so the line
|
||||
appears and offers the recorded view alone. With one, the arrows and the depth map
|
||||
are two file reads away, so the line is gone — and it is gone because the modes are
|
||||
on offer, not because a message was suppressed.
|
||||
"""
|
||||
raw = ImageFrame(np.zeros((8, 8, 3), np.uint8), source=SOURCE_REPLAY)
|
||||
plan = plan_image(raw)
|
||||
assert plan.available_modes == ("recorded",)
|
||||
assert plan.note_key == RAW_NOTE_KEY
|
||||
assert plan.resolve(DEPTH) == "recorded", "a view that cannot be drawn was selected"
|
||||
assert plan.has_depth() is False and plan.has_flow() is False
|
||||
|
||||
backed = ImageFrame(
|
||||
np.zeros((8, 8, 3), np.uint8), source=SOURCE_REPLAY, result_available=True
|
||||
)
|
||||
plan = plan_image(backed)
|
||||
assert plan.note_key == "", "the pane still says there is nothing to draw"
|
||||
assert plan.available_modes == ("recorded", *DERIVED_MODES)
|
||||
assert plan.default_mode == "recorded", "the raw picture stopped being the default"
|
||||
for mode in DERIVED_MODES:
|
||||
assert plan.accepts(mode), f"{mode} is not on offer with a result in hand"
|
||||
|
||||
|
||||
def test_the_nothing_to_draw_line_names_the_action_that_produces_a_result():
|
||||
"""Both locales: the line says what to do, not merely that there is nothing."""
|
||||
from orisys_app_shared.i18n import set_locale
|
||||
|
||||
seen = {}
|
||||
try:
|
||||
for locale in ("zh_CN", "en_US"):
|
||||
set_locale(locale)
|
||||
text = t(RAW_NOTE_KEY)
|
||||
seen[locale] = text
|
||||
# The action that produces a result is named: the pass runs on open, and the
|
||||
# toolbar action re-runs it.
|
||||
assert "由视频重算" in text or "Recompute from the video" in text, text
|
||||
assert (
|
||||
"分析" in text or "analysis result" in text
|
||||
), f"{locale} does not say a result is what is missing: {text}"
|
||||
finally:
|
||||
set_locale("zh_CN")
|
||||
assert seen["zh_CN"] != seen["en_US"]
|
||||
|
||||
|
||||
def test_the_empty_depth_note_is_not_the_missing_depth_note():
|
||||
"""Both locales keep the two apart: a flat map is a reading, an absent one is not."""
|
||||
from orisys_app_shared.i18n import set_locale
|
||||
|
||||
try:
|
||||
for locale in ("zh_CN", "en_US"):
|
||||
set_locale(locale)
|
||||
empty = t("panels.depth_empty", pixels=84000)
|
||||
absent = t("panels.depth_unavailable")
|
||||
assert "84000" in empty or "84,000" in empty, empty
|
||||
assert empty != absent
|
||||
finally:
|
||||
set_locale("zh_CN")
|
||||
|
||||
|
||||
def test_a_derived_mode_is_served_from_the_result_even_on_the_recording_timeline(
|
||||
pass_root,
|
||||
):
|
||||
"""Through the controller: the depth mode draws the stored frame, not the video.
|
||||
|
||||
The timeline is put on the recording (which is the state the operator is in while a
|
||||
pass runs, and the state a chosen source leaves them in), and the mode selector is
|
||||
set to depth. What reaches the window's single 2D sink must be the result's own frame
|
||||
— with its depth map — and not the decoded video frame, which carries none.
|
||||
"""
|
||||
_require_inputs()
|
||||
reader = AnalysisReader(_result_directory(pass_root))
|
||||
session = AnalysisSession(
|
||||
reader,
|
||||
display_policy=policy_for_settings(DisplaySettings()),
|
||||
# The recording the result describes. ``available_result`` matches it against the
|
||||
# recording that is open, which is how a previous recording's result cannot
|
||||
# answer for this one.
|
||||
recording=RECORDING,
|
||||
)
|
||||
try:
|
||||
panel = _ImagePanel()
|
||||
controller, host, engine, _ = _controller(image_panel=panel)
|
||||
controller.results.open_session(session)
|
||||
engine.use_analysis(session)
|
||||
engine._mode = MODE_RECORDING # the timeline reads the video; the view does not
|
||||
engine.set_position(session.time_of(FIRST_FRAME_ID + 1))
|
||||
host.frames.clear()
|
||||
|
||||
panel.set_view_mode(DEPTH)
|
||||
controller.render_frame()
|
||||
assert host.frames, "the depth mode drew nothing"
|
||||
drawn = host.frames[-1]
|
||||
assert drawn.source == SOURCE_REPLAY, "the recording timeline was not kept"
|
||||
assert drawn.result_available is True
|
||||
stored = session.frame(FIRST_FRAME_ID + 1)
|
||||
assert drawn.frame_id == stored.frame_id
|
||||
assert np.array_equal(np.asarray(drawn.depth), np.asarray(stored.depth))
|
||||
assert drawn.depth_valid is True
|
||||
assert plan_image(drawn).note_key == "", "the pane despaired with a result in hand"
|
||||
|
||||
# And the raw view still shows the recording's own picture.
|
||||
panel.set_view_mode("recorded")
|
||||
host.frames.clear()
|
||||
controller.render_frame()
|
||||
assert host.frames[-1].depth is None
|
||||
assert host.frames[-1].source == SOURCE_REPLAY
|
||||
|
||||
# Now through the engine's own way back to the video: it forgets the session it
|
||||
# was borrowing, and the *library* still holds the result for this recording. The
|
||||
# derived mode must keep drawing from it — a result is a property of the
|
||||
# recording, not of which timeline the cursor is on.
|
||||
engine.use_recording()
|
||||
assert engine.analysis() is None
|
||||
panel.set_view_mode(DEPTH)
|
||||
host.frames.clear()
|
||||
controller.render_frame()
|
||||
assert host.frames[-1].depth is not None, "the depth view followed the borrow"
|
||||
assert np.array_equal(
|
||||
np.asarray(host.frames[-1].depth), np.asarray(stored.depth)
|
||||
)
|
||||
assert plan_image(host.frames[-1]).note_key == ""
|
||||
finally:
|
||||
reader.close()
|
||||
|
||||
@@ -285,6 +285,10 @@ class _Engine:
|
||||
def video_message(self) -> str:
|
||||
return "no frame"
|
||||
|
||||
def analysis(self):
|
||||
"""No stored result in this stub: the derived views fall back to the video."""
|
||||
return None
|
||||
|
||||
def set_unit(self, unit: str) -> None:
|
||||
self.units.append(str(unit))
|
||||
|
||||
@@ -411,7 +415,6 @@ def test_a_bare_video_is_opened_as_a_video_and_refused_outside_the_view(qt_app,
|
||||
bare.write_bytes(b"not really a video")
|
||||
engine = _Engine()
|
||||
controller, host, engine, parts = _controller(engine=engine)
|
||||
toolbar = parts[0]
|
||||
|
||||
host.view = "live"
|
||||
assert controller.load_video(bare) is False
|
||||
|
||||
@@ -186,6 +186,13 @@ class ImageFrame:
|
||||
flow_frame_id: int | None = None
|
||||
depth_valid: bool | None = None
|
||||
depth_frame_id: int | None = None
|
||||
#: True when a stored **analysis result** exists for this recording, so the derived
|
||||
#: views have a frame of their own to draw even when *this* frame is the recording's
|
||||
#: raw picture. It is what keeps the panel from telling the operator there is nothing
|
||||
#: to draw while a result is sitting on disk: the modes are offered, and the view
|
||||
#: that needs derived data is served from the result. ``False`` means no result — the
|
||||
#: only case in which "a recording stores only the raw camera image" is the truth.
|
||||
result_available: bool = False
|
||||
|
||||
def has_flow(self) -> bool:
|
||||
if self.flow is None:
|
||||
@@ -243,16 +250,23 @@ class ImagePlan:
|
||||
|
||||
|
||||
def plan_image(frame: ImageFrame) -> ImagePlan:
|
||||
"""Derive the modes from the frame's *source* and its own fields.
|
||||
"""Derive the modes from the frame's *source*, its own fields, and any result behind it.
|
||||
|
||||
Two different questions, deliberately answered in two different places:
|
||||
|
||||
* **which views may the operator choose?** For a source that carries derived fields
|
||||
(live, recompute) all of them — a single dropped frame must not silently switch
|
||||
the operator's view. For a source that never does (a recording, a camera preview)
|
||||
exactly one, because offering the others would be offering a lie;
|
||||
(live, recompute, analysis) all of them — a single dropped frame must not silently
|
||||
switch the operator's view. A recording offers them too *when a stored analysis
|
||||
result exists for it* (``result_available``): the recording's own picture carries
|
||||
no flow, but the result does, and the view that needs it is drawn from the result.
|
||||
With no result there is nothing behind the picture at all, so the raw frame is the
|
||||
only thing it can honestly be;
|
||||
* **what can this frame actually be drawn as?** ``missing`` says which field is
|
||||
absent, and the panel states it in one line.
|
||||
|
||||
The raw-recording note is therefore an *absence* statement and appears only in the
|
||||
one case where it is true: no result, nothing to draw, and the panel says which
|
||||
action produces one.
|
||||
"""
|
||||
derived = str(frame.source or SOURCE_LIVE) in DERIVED_SOURCES
|
||||
missing: list[str] = []
|
||||
@@ -260,7 +274,7 @@ def plan_image(frame: ImageFrame) -> ImagePlan:
|
||||
missing.append("flow")
|
||||
if not frame.has_depth():
|
||||
missing.append("depth")
|
||||
if not derived:
|
||||
if not derived and not frame.result_available:
|
||||
return ImagePlan(
|
||||
available_modes=(RECORDED,),
|
||||
default_mode=RECORDED,
|
||||
@@ -271,6 +285,19 @@ def plan_image(frame: ImageFrame) -> ImagePlan:
|
||||
stale=bool(frame.stale),
|
||||
note_key=RAW_NOTE_KEY,
|
||||
)
|
||||
if not derived:
|
||||
# A recording with a result waiting behind it. ``RECORDED`` stays the default —
|
||||
# the picture on screen *is* the recorded image until the operator asks for
|
||||
# something else — and nothing is claimed to be missing, because nothing is.
|
||||
return ImagePlan(
|
||||
available_modes=(RECORDED, *DERIVED_MODES),
|
||||
default_mode=RECORDED,
|
||||
missing=(),
|
||||
source=str(frame.source or SOURCE_LIVE),
|
||||
frame_id=int(frame.frame_id),
|
||||
t_rel_s=float(frame.t_rel_s),
|
||||
stale=bool(frame.stale),
|
||||
)
|
||||
default = ARROW_PURE if frame.has_flow() else (DEPTH if frame.has_depth() else ARROW_PURE)
|
||||
return ImagePlan(
|
||||
available_modes=DERIVED_MODES,
|
||||
|
||||
@@ -7,8 +7,10 @@ and stores the flow samples a replay draws, so seeking is a file read.
|
||||
|
||||
What is stored is deliberately *not* the chain's output. One frame of raw output is
|
||||
1,176,000 B (``flow`` [300,280,2] float32, ``depth_map``, ``valid_mask``, the gray
|
||||
image), which is 21.2 GB for ten minutes at 30 fps. The selected samples cost 4,034 B
|
||||
per frame. The rest is recomputed — that is what makes the cache a cache.
|
||||
image), which is 21.2 GB for ten minutes at 30 fps. The selected samples cost 4,048 B
|
||||
per frame, and the depth channel — which is all zeros on the great majority of frames —
|
||||
costs 14 B more when it is empty. The rest is recomputed — that is what makes the cache
|
||||
a cache.
|
||||
|
||||
The three modules have one job each:
|
||||
|
||||
|
||||
@@ -10,8 +10,9 @@ properties matter more than speed:
|
||||
pass leaves a directory that refuses to open by default.
|
||||
* **Memory must not grow with the recording.** Frames accumulate into a fixed-size
|
||||
chunk and the chunk is written once. The bound is
|
||||
``chunk_frames * frame_payload_bytes(max candidates)`` — 256 frames of the shipped
|
||||
finger grid is about 1.0 MB — and ``chunk_frames`` is clamped so a caller cannot
|
||||
``chunk_frames * frame_payload_bytes(max candidates, with_depth=True)`` plus one
|
||||
depth plane per frame that carries a non-empty one — 256 frames of the shipped
|
||||
finger grid is about 1.1 MB — and ``chunk_frames`` is clamped so a caller cannot
|
||||
raise it without bound.
|
||||
|
||||
Storage is not free, so :func:`evict` removes whole results against a stated size cap.
|
||||
|
||||
@@ -5,7 +5,7 @@ only what a replay draws: the flow samples the shared selector returns, not the
|
||||
field they came from. The fields a frame produces are far too large to cache — the
|
||||
measured cost is 1,176,000 B per frame (``flow`` 672,000 + ``depth_map`` 336,000 +
|
||||
``valid_mask`` 84,000 + the gray ``img`` 84,000), which is 21.2 GB for ten minutes at
|
||||
30 fps. The samples themselves cost 4,034 B per frame at the shipped finger grid.
|
||||
30 fps. The samples themselves cost 4,048 B per frame at the shipped finger grid.
|
||||
|
||||
Three structural facts decide the layout, and each one is a measured result rather
|
||||
than a guess:
|
||||
@@ -28,6 +28,28 @@ served by re-deriving activity from magnitude; only a change to the sample *grid
|
||||
(``flow_grid_spacing``, ``flow_edge_margin``, ``max_arrows``) moves the candidate
|
||||
locations, and that policy is recorded in the provenance for comparison instead of
|
||||
being baked into the key.
|
||||
|
||||
**Depth is stored, and it is stored quantised.** A frame's own ``depth_map`` is
|
||||
336,000 B (300 × 280 float32) — 6.0 GB per ten minutes at 30 fps, which is why the
|
||||
obvious encoding is out — and on the two real recordings it is empty far more often
|
||||
than not: 241 of 264 frames of the eight-times clip and 32 of 33 frames of the box
|
||||
recording carry a map that is *exactly* all zeros, with the worst non-empty frame at
|
||||
336 of 84,000 pixels. So the record states the depth channel's own statistics every
|
||||
frame (present, non-zero count, minimum, maximum) and appends a payload only for a
|
||||
frame that has one, as uint8 over that frame's own range, zlib'd at level 6. The
|
||||
measured cost is 14 B per empty frame and 486 B for the worst non-empty frame seen;
|
||||
the largest error that can be introduced is half a quantisation step,
|
||||
``(max - min) / 510``, measured at 6.6e-5 on the clip whose peak is 0.034. That is
|
||||
below half of one step of the 256-entry colour ramp the depth view paints with, so a
|
||||
reconstructed frame cannot be coloured differently from the original.
|
||||
|
||||
Sparse storage was measured and rejected: 4-byte indices per non-zero pixel cost
|
||||
910 B for the worst non-empty frame against 486 B for the whole quantised map, because
|
||||
a mostly-zero byte plane is nearly free for zlib while pixel indices are not. Island
|
||||
restriction was measured and rejected too — 418 B against 486 B, a saving of 5.2 B per
|
||||
frame of a 4,048 B record — because it would make the reconstruction depend on the
|
||||
validity-mask keyframe (the box chain has none) and would silently drop a depth value
|
||||
that fell outside the island. See ``tests/test_analysis_depth.py`` for the numbers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -35,6 +57,7 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import json
|
||||
import struct
|
||||
import zlib
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
@@ -50,8 +73,12 @@ __all__ = [
|
||||
"CHUNK_HEADER",
|
||||
"CHUNK_MAGIC",
|
||||
"CHUNK_TRAILER",
|
||||
"CODEC_EMPTY",
|
||||
"CODEC_RAW",
|
||||
"CODEC_ZLIB",
|
||||
"DEPTH_BITS",
|
||||
"DEPTH_HEADER",
|
||||
"FRAME_FLAG_DEPTH",
|
||||
"FRAME_FLAG_DROPPED",
|
||||
"FRAME_FLAG_VALIDITY",
|
||||
"FRAME_HEADER",
|
||||
@@ -59,12 +86,16 @@ __all__ = [
|
||||
"KEYFRAME_HEADER",
|
||||
"KEYFRAME_MAGIC",
|
||||
"KEYFRAME_MASK",
|
||||
"MAX_DEPTH_PIXELS",
|
||||
"UNIT",
|
||||
"DecodedDepth",
|
||||
"FramePayload",
|
||||
"KeyframeRef",
|
||||
"Provenance",
|
||||
"ResultKey",
|
||||
"decode_depth",
|
||||
"decode_frame",
|
||||
"encode_depth",
|
||||
"encode_frame",
|
||||
"frame_payload_bytes",
|
||||
"keyframe_payload",
|
||||
@@ -78,8 +109,9 @@ RESULT_DIRNAME = "analysis"
|
||||
|
||||
#: On-disk result format. Bump only for an incompatible layout change; a change to
|
||||
#: what the *chain computes* belongs in :data:`ALGORITHM_VERSION` instead.
|
||||
#: Version 2 adds the depth channel to the frame record (see the module docstring).
|
||||
ANALYSIS_FORMAT = "orisys-replay-analysis"
|
||||
ANALYSIS_FORMAT_VERSION = 1
|
||||
ANALYSIS_FORMAT_VERSION = 2
|
||||
|
||||
#: Identity of the analysis the pass runs: the SDK calls it makes, in order.
|
||||
#: ``compute_deformation(decompose=True, check_motion=False)`` then
|
||||
@@ -92,11 +124,16 @@ ALGORITHM_VERSION = "flow-hhd-contact-1"
|
||||
#: unit :class:`~orisys_app_shared.scene3d.visual_policy.FlowSamples` documents.
|
||||
UNIT = "uv_px"
|
||||
|
||||
#: ``frame_id``, ``t_rel_s``, grid keyframe id, flags, candidate count, depth statistic.
|
||||
#: The four trailing pad bytes keep the value arrays 8-byte aligned inside a record.
|
||||
FRAME_HEADER = struct.Struct("<idHBHI3x")
|
||||
#: ``frame_id``, ``t_rel_s``, grid keyframe id, flags, candidate count, depth
|
||||
#: non-zero count, depth minimum, depth maximum. The three trailing pad bytes keep
|
||||
#: the value arrays 8-byte aligned inside a record.
|
||||
FRAME_HEADER = struct.Struct("<idHBHIff3x")
|
||||
FRAME_FLAG_VALIDITY = 0x01
|
||||
FRAME_FLAG_DROPPED = 0x02
|
||||
#: The chain exposed a depth map for this frame: the three depth statistics in the
|
||||
#: header describe it and a depth payload follows. Absent flag means *no depth
|
||||
#: channel at all*, which is a different fact from "the channel was all zeros".
|
||||
FRAME_FLAG_DEPTH = 0x04
|
||||
|
||||
#: magic, kind, codec, count, height, width, flags, reserved, raw bytes, digest
|
||||
KEYFRAME_HEADER = struct.Struct("<4sBBIIIBBQ8s")
|
||||
@@ -104,8 +141,22 @@ KEYFRAME_MAGIC = b"ORAK"
|
||||
KEYFRAME_GRID = 0
|
||||
KEYFRAME_MASK = 1
|
||||
|
||||
#: codec, bits per stored value, height, width. A depth payload is this header
|
||||
#: followed by its stored bytes — none at all for :data:`CODEC_EMPTY`.
|
||||
DEPTH_HEADER = struct.Struct("<BBHH")
|
||||
DEPTH_BITS = 8
|
||||
|
||||
CODEC_RAW = 0
|
||||
CODEC_ZLIB = 1
|
||||
#: The map is exactly all zeros: ``height × width`` of them, and no payload. This is
|
||||
#: the common case by a wide margin, and saying so explicitly is what lets a reader
|
||||
#: explain an empty depth view instead of drawing a blank one.
|
||||
CODEC_EMPTY = 2
|
||||
|
||||
#: Refuse a depth header that asks for an implausible array before allocating it. The
|
||||
#: shipped grid is 300 × 280; this leaves room for a far larger sensor and no room for
|
||||
#: a corrupt header to become a 4 GB allocation.
|
||||
MAX_DEPTH_PIXELS = 1 << 26
|
||||
|
||||
#: magic, version, reserved, chunk index, frame count, header size
|
||||
CHUNK_HEADER = struct.Struct("<4sHHIII")
|
||||
@@ -207,6 +258,188 @@ def recording_identity(
|
||||
return canonical_digest8(payload).hex()
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- depth channel
|
||||
def _depth_plane(depth: Any) -> np.ndarray:
|
||||
"""A depth map as a 2-D float32 array, whatever leading singletons it arrives with.
|
||||
|
||||
The finger chain hands ``depth_map`` over as ``(1, H, W)`` — a batch axis nobody
|
||||
batches — and the validity mask it belongs beside is ``(H, W)``. Squeezing only
|
||||
leading singletons keeps a genuinely 3-D field (a future multi-plane depth) an
|
||||
error rather than something silently reinterpreted.
|
||||
"""
|
||||
array = np.asarray(depth, dtype=np.float32)
|
||||
while array.ndim > 2 and array.shape[0] == 1:
|
||||
array = array[0]
|
||||
if array.ndim != 2:
|
||||
raise ValueError(f"a depth map is 2-D, got shape {np.shape(depth)}")
|
||||
return np.ascontiguousarray(array)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DecodedDepth:
|
||||
"""One frame's depth channel: how it was stored, and the statistics that describe it."""
|
||||
|
||||
codec: int
|
||||
height: int
|
||||
width: int
|
||||
nonzero: int
|
||||
minimum: float
|
||||
maximum: float
|
||||
#: The stored bytes below the header — empty for :data:`CODEC_EMPTY`, and the
|
||||
#: quantised plane decompressed for the other codecs.
|
||||
stored: bytes = b""
|
||||
#: How many bytes those were on disk, before decompression. Kept separate from
|
||||
#: ``len(stored)`` because the compressed and the decoded length are different
|
||||
#: numbers and only one of them is what the record costs.
|
||||
stored_bytes: int = 0
|
||||
|
||||
@property
|
||||
def pixels(self) -> int:
|
||||
return int(self.height) * int(self.width)
|
||||
|
||||
@property
|
||||
def payload_bytes(self) -> int:
|
||||
"""Every byte this frame's depth channel occupies, header included."""
|
||||
return DEPTH_HEADER.size + int(self.stored_bytes)
|
||||
|
||||
@property
|
||||
def is_empty(self) -> bool:
|
||||
return self.codec == CODEC_EMPTY
|
||||
|
||||
|
||||
def _quantise(depth: np.ndarray) -> tuple[np.ndarray, float, float]:
|
||||
"""uint8 codes over the array's own ``[min, max]``, plus that range.
|
||||
|
||||
The range is the frame's own rather than a session-wide one on purpose: a depth
|
||||
response that is large on one frame and small on the next keeps its full 256 levels
|
||||
either way. The price is that the range has to travel with the frame, which is why
|
||||
it lives in the frame header (8 B) instead of in each payload.
|
||||
"""
|
||||
low = float(depth.min())
|
||||
high = float(depth.max())
|
||||
span = high - low
|
||||
if span <= 0.0:
|
||||
return np.zeros(depth.shape, dtype=np.uint8), low, high
|
||||
scaled = (depth.astype(np.float64) - low) / span
|
||||
codes = np.clip(np.rint(scaled * 255.0), 0.0, 255.0).astype(np.uint8)
|
||||
return codes, low, high
|
||||
|
||||
|
||||
def _dequantise(codes: np.ndarray, low: float, high: float) -> np.ndarray:
|
||||
span = float(high) - float(low)
|
||||
if span <= 0.0:
|
||||
return np.full(codes.shape, float(low), dtype=np.float32)
|
||||
return (float(low) + codes.astype(np.float64) * (span / 255.0)).astype(np.float32)
|
||||
|
||||
|
||||
def encode_depth(depth: Any) -> tuple[bytes, int, float, float]:
|
||||
"""One frame's depth channel as ``(payload, nonzero, min, max)``.
|
||||
|
||||
The payload is a :data:`DEPTH_HEADER` and, unless the map is exactly all zeros, the
|
||||
zlib'd uint8 plane. An empty map stores the header alone: 6 bytes that state the
|
||||
shape, which is what a reader needs to rebuild the zeros rather than guess at them.
|
||||
"""
|
||||
array = _depth_plane(depth)
|
||||
height, width = int(array.shape[0]), int(array.shape[1])
|
||||
if height * width > MAX_DEPTH_PIXELS:
|
||||
raise ValueError(f"a {height}x{width} depth map is larger than this format holds")
|
||||
nonzero = int(np.count_nonzero(array))
|
||||
low = float(array.min())
|
||||
high = float(array.max())
|
||||
if nonzero == 0:
|
||||
return (
|
||||
DEPTH_HEADER.pack(CODEC_EMPTY, DEPTH_BITS, height, width),
|
||||
nonzero,
|
||||
low,
|
||||
high,
|
||||
)
|
||||
codes, low, high = _quantise(array)
|
||||
raw = codes.tobytes()
|
||||
stored = zlib.compress(raw, 6)
|
||||
codec = CODEC_ZLIB
|
||||
if len(stored) >= len(raw):
|
||||
codec = CODEC_RAW
|
||||
stored = raw
|
||||
return (
|
||||
DEPTH_HEADER.pack(codec, DEPTH_BITS, height, width) + stored,
|
||||
nonzero,
|
||||
low,
|
||||
high,
|
||||
)
|
||||
|
||||
|
||||
def decode_depth(
|
||||
buffer: bytes | bytearray | memoryview,
|
||||
offset: int = 0,
|
||||
*,
|
||||
low: float = 0.0,
|
||||
high: float = 0.0,
|
||||
) -> DecodedDepth:
|
||||
"""Read one depth payload. Raises :class:`ValueError` on anything inconsistent.
|
||||
|
||||
The stored plane is checked against the shape its own header declares, so a
|
||||
truncated or mis-sized payload is refused here rather than surfacing later as an
|
||||
array of the wrong shape.
|
||||
"""
|
||||
view = memoryview(buffer)
|
||||
if len(view) < offset + DEPTH_HEADER.size:
|
||||
raise ValueError("a depth payload is shorter than its header")
|
||||
codec, bits, height, width = DEPTH_HEADER.unpack_from(view, offset)
|
||||
cursor = offset + DEPTH_HEADER.size
|
||||
pixels = int(height) * int(width)
|
||||
if pixels <= 0 or pixels > MAX_DEPTH_PIXELS:
|
||||
raise ValueError(f"a depth payload declares an implausible shape {height}x{width}")
|
||||
if int(bits) != DEPTH_BITS:
|
||||
raise ValueError(f"a depth payload stores {bits} bits per value, not {DEPTH_BITS}")
|
||||
stored = bytes(view[cursor:])
|
||||
if int(codec) == CODEC_EMPTY:
|
||||
return DecodedDepth(
|
||||
codec=int(codec),
|
||||
height=int(height),
|
||||
width=int(width),
|
||||
nonzero=0,
|
||||
minimum=0.0,
|
||||
maximum=0.0,
|
||||
)
|
||||
if int(codec) not in (CODEC_RAW, CODEC_ZLIB):
|
||||
raise ValueError(f"a depth payload uses codec {codec}")
|
||||
if int(codec) == CODEC_ZLIB:
|
||||
try:
|
||||
raw = zlib.decompress(stored)
|
||||
except zlib.error as exc:
|
||||
raise ValueError(f"a depth payload does not decompress: {exc}") from exc
|
||||
else:
|
||||
raw = stored
|
||||
if len(raw) != pixels:
|
||||
raise ValueError(
|
||||
f"a depth payload decoded to {len(raw)} bytes for a {height}x{width} map"
|
||||
)
|
||||
return DecodedDepth(
|
||||
codec=int(codec),
|
||||
height=int(height),
|
||||
width=int(width),
|
||||
nonzero=pixels,
|
||||
minimum=float(low),
|
||||
maximum=float(high),
|
||||
stored=raw,
|
||||
stored_bytes=len(stored),
|
||||
)
|
||||
|
||||
|
||||
def depth_values(decoded: DecodedDepth, *, low: float, high: float) -> np.ndarray:
|
||||
"""The depth map a decoded payload describes, as float32.
|
||||
|
||||
A :data:`CODEC_EMPTY` payload becomes ``height × width`` exact zeros — the map the
|
||||
chain produced, not a placeholder — so the depth view and its statistics describe
|
||||
the same array.
|
||||
"""
|
||||
shape = (int(decoded.height), int(decoded.width))
|
||||
if decoded.codec == CODEC_EMPTY:
|
||||
return np.zeros(shape, dtype=np.float32)
|
||||
codes = np.frombuffer(decoded.stored, dtype=np.uint8).reshape(shape)
|
||||
return _dequantise(codes, float(low), float(high))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- frame encoding
|
||||
def encode_frame(
|
||||
*,
|
||||
@@ -218,15 +451,22 @@ def encode_frame(
|
||||
magnitude: np.ndarray,
|
||||
active: np.ndarray,
|
||||
validity_digest: bytes | None = None,
|
||||
depth: Any | None = None,
|
||||
depth_nonzero: int = 0,
|
||||
dropped: bool = False,
|
||||
) -> bytes:
|
||||
"""One frame record: a fixed header, then the value arrays.
|
||||
"""One frame record: a fixed header, then the value arrays and the depth payload.
|
||||
|
||||
Arrays are little-endian regardless of host order, so a result written on one
|
||||
machine reads back identically on another. ``dx``/``dy``/``magnitude`` are stored
|
||||
as float32 — the precision the display needs and the reason a stored payload is
|
||||
compared to a live selection *cast to float32* rather than in float64.
|
||||
|
||||
``depth`` is the frame's own map, or ``None`` when the chain exposed none; the two
|
||||
are different facts (``FRAME_FLAG_DEPTH``) and the header's statistics belong to the
|
||||
first. ``depth_nonzero`` is only consulted when ``depth`` is ``None``, which is how
|
||||
a caller that can count but cannot afford to hand the array over still records the
|
||||
count exactly.
|
||||
"""
|
||||
dx = np.ascontiguousarray(dx, dtype="<f4").reshape(-1)
|
||||
dy = np.ascontiguousarray(dy, dtype="<f4").reshape(-1)
|
||||
@@ -247,6 +487,11 @@ def encode_frame(
|
||||
flags |= FRAME_FLAG_VALIDITY
|
||||
if dropped:
|
||||
flags |= FRAME_FLAG_DROPPED
|
||||
depth_payload = b""
|
||||
depth_low = depth_high = 0.0
|
||||
if depth is not None:
|
||||
flags |= FRAME_FLAG_DEPTH
|
||||
depth_payload, depth_nonzero, depth_low, depth_high = encode_depth(depth)
|
||||
parts = [
|
||||
FRAME_HEADER.pack(
|
||||
int(frame_id),
|
||||
@@ -255,6 +500,8 @@ def encode_frame(
|
||||
flags,
|
||||
count,
|
||||
max(0, int(depth_nonzero)),
|
||||
float(depth_low),
|
||||
float(depth_high),
|
||||
),
|
||||
dx.tobytes(),
|
||||
dy.tobytes(),
|
||||
@@ -266,6 +513,7 @@ def encode_frame(
|
||||
if len(digest) != 8:
|
||||
raise ValueError(f"validity digest must be 8 bytes, got {len(digest)}")
|
||||
parts.append(digest)
|
||||
parts.append(depth_payload)
|
||||
return b"".join(parts)
|
||||
|
||||
|
||||
@@ -284,6 +532,17 @@ class DecodedFrame:
|
||||
magnitude: np.ndarray
|
||||
active: np.ndarray
|
||||
validity_digest: bytes | None
|
||||
#: ``None`` when the frame carried no depth channel; otherwise the payload, whose
|
||||
#: ``stored`` bytes are still quantised and need :func:`depth_values`.
|
||||
depth: DecodedDepth | None = None
|
||||
|
||||
def depth_map(self) -> np.ndarray | None:
|
||||
"""This frame's depth map, or ``None`` when it had no depth channel."""
|
||||
if self.depth is None:
|
||||
return None
|
||||
return depth_values(
|
||||
self.depth, low=self.depth.minimum, high=self.depth.maximum
|
||||
)
|
||||
|
||||
|
||||
def decode_frame(
|
||||
@@ -291,9 +550,16 @@ def decode_frame(
|
||||
) -> DecodedFrame:
|
||||
"""Decode one frame record starting at ``offset``. Ignores any trailing bytes."""
|
||||
view = memoryview(buffer)
|
||||
frame_id, t_rel, grid_id, flags, count, depth_nonzero = FRAME_HEADER.unpack_from(
|
||||
view, offset
|
||||
)
|
||||
(
|
||||
frame_id,
|
||||
t_rel,
|
||||
grid_id,
|
||||
flags,
|
||||
count,
|
||||
depth_nonzero,
|
||||
depth_low,
|
||||
depth_high,
|
||||
) = FRAME_HEADER.unpack_from(view, offset)
|
||||
cursor = offset + FRAME_HEADER.size
|
||||
arrays = []
|
||||
for _ in range(3):
|
||||
@@ -308,6 +574,12 @@ def decode_frame(
|
||||
validity = None
|
||||
if flags & FRAME_FLAG_VALIDITY:
|
||||
validity = bytes(view[cursor : cursor + 8])
|
||||
cursor += 8
|
||||
depth = None
|
||||
if flags & FRAME_FLAG_DEPTH:
|
||||
depth = decode_depth(
|
||||
view, cursor, low=float(depth_low), high=float(depth_high)
|
||||
)
|
||||
return DecodedFrame(
|
||||
frame_id=int(frame_id),
|
||||
t_rel_s=float(t_rel),
|
||||
@@ -320,13 +592,29 @@ def decode_frame(
|
||||
magnitude=arrays[2],
|
||||
active=active.astype(bool),
|
||||
validity_digest=validity,
|
||||
depth=depth,
|
||||
)
|
||||
|
||||
|
||||
def frame_payload_bytes(candidates: int, *, with_validity: bool = True) -> int:
|
||||
"""Stored bytes of one frame record at ``candidates`` sample locations."""
|
||||
def frame_payload_bytes(
|
||||
candidates: int,
|
||||
*,
|
||||
with_validity: bool = True,
|
||||
with_depth: bool = False,
|
||||
depth_stored: int = 0,
|
||||
) -> int:
|
||||
"""Stored bytes of one frame record at ``candidates`` sample locations.
|
||||
|
||||
``with_depth`` adds the 6-byte depth header every frame that carries a channel pays;
|
||||
``depth_stored`` adds a payload of that many bytes on top, for a caller sizing the
|
||||
memory bound rather than the common case.
|
||||
"""
|
||||
total = FRAME_HEADER.size + int(candidates) * 12 + (int(candidates) + 7) // 8
|
||||
return total + 8 if with_validity else total
|
||||
if with_validity:
|
||||
total += 8
|
||||
if with_depth:
|
||||
total += DEPTH_HEADER.size + max(0, int(depth_stored))
|
||||
return total
|
||||
|
||||
|
||||
def keyframe_payload(kind: int, array: np.ndarray) -> bytes:
|
||||
@@ -373,6 +661,30 @@ class FramePayload:
|
||||
validity_digest: bytes | None = None
|
||||
depth_nonzero: int = 0
|
||||
dropped: bool = False
|
||||
#: This frame's depth map at the chain's own resolution, or ``None`` when the frame
|
||||
#: carried no depth channel. An all-zero map is *not* ``None``: "the response was
|
||||
#: zero everywhere" and "there was no response" are different answers and a depth
|
||||
#: view that shows the same blank picture for both is the one that misleads.
|
||||
depth: np.ndarray | None = None
|
||||
depth_min: float = 0.0
|
||||
depth_max: float = 0.0
|
||||
#: Stored bytes of this frame's depth payload — the 6-byte header for an empty map,
|
||||
#: header plus plane otherwise, and 0 when the frame had no depth channel. It is
|
||||
#: what a caller adds to the fixed part of a record to say what depth cost.
|
||||
depth_stored_bytes: int = 0
|
||||
|
||||
@property
|
||||
def has_depth(self) -> bool:
|
||||
return self.depth is not None
|
||||
|
||||
@property
|
||||
def depth_is_empty(self) -> bool:
|
||||
"""True when a depth channel exists and is exactly zero everywhere."""
|
||||
return self.depth is not None and self.depth_nonzero == 0
|
||||
|
||||
@property
|
||||
def depth_pixels(self) -> int:
|
||||
return 0 if self.depth is None else int(self.depth.size)
|
||||
|
||||
@property
|
||||
def candidates(self) -> int:
|
||||
@@ -412,6 +724,7 @@ class FramePayload:
|
||||
magnitude=self.magnitude,
|
||||
active=self.active,
|
||||
validity_digest=self.validity_digest,
|
||||
depth=self.depth,
|
||||
depth_nonzero=self.depth_nonzero,
|
||||
dropped=self.dropped,
|
||||
)
|
||||
|
||||
@@ -462,6 +462,25 @@ class AnalysisReader:
|
||||
f"frame {frame_id} has {decoded.count} values but grid "
|
||||
f"{decoded.grid_id} has {rows.size} locations"
|
||||
)
|
||||
depth = None
|
||||
depth_min = depth_max = 0.0
|
||||
depth_stored = 0
|
||||
if decoded.depth is not None:
|
||||
try:
|
||||
depth = decoded.depth_map()
|
||||
except ValueError as exc:
|
||||
raise AnalysisFormatError(f"frame {frame_id}: {exc}") from exc
|
||||
depth_min = float(decoded.depth.minimum)
|
||||
depth_max = float(decoded.depth.maximum)
|
||||
depth_stored = decoded.depth.payload_bytes
|
||||
counted = int(np.count_nonzero(depth))
|
||||
if counted != decoded.depth_nonzero:
|
||||
# The header's count and the stored plane must describe one array. A
|
||||
# mismatch means the record is corrupt, not that the map is empty.
|
||||
raise AnalysisFormatError(
|
||||
f"frame {frame_id} stores {decoded.depth_nonzero} non-zero depth "
|
||||
f"pixels but its payload holds {counted}"
|
||||
)
|
||||
return FramePayload(
|
||||
frame_id=decoded.frame_id,
|
||||
t_rel_s=decoded.t_rel_s,
|
||||
@@ -475,6 +494,10 @@ class AnalysisReader:
|
||||
validity_digest=decoded.validity_digest,
|
||||
depth_nonzero=decoded.depth_nonzero,
|
||||
dropped=decoded.dropped,
|
||||
depth=depth,
|
||||
depth_min=depth_min,
|
||||
depth_max=depth_max,
|
||||
depth_stored_bytes=depth_stored,
|
||||
)
|
||||
|
||||
def frame_at(self, t_rel_s: float) -> FramePayload | None:
|
||||
|
||||
@@ -371,6 +371,9 @@ class LiveFrame:
|
||||
depth_abs_sum: float
|
||||
reset_applied: bool
|
||||
segment: int
|
||||
#: The chain's depth map for this frame, or ``None`` when it exposed none. Held only
|
||||
#: until the writer has encoded it: one frame at a time, never accumulated.
|
||||
depth: np.ndarray | None = None
|
||||
|
||||
def payload_matches(self, stored: Any) -> bool:
|
||||
"""True when a stored payload equals this frame's selection, in stored precision.
|
||||
@@ -836,6 +839,7 @@ class AnalysisService:
|
||||
),
|
||||
reset_applied=reset_applied,
|
||||
segment=int(number),
|
||||
depth=depth_array,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------- internals
|
||||
@@ -927,9 +931,17 @@ class AnalysisService:
|
||||
magnitude=samples.magnitude,
|
||||
active=samples.active,
|
||||
validity_digest=validity_digest,
|
||||
depth=live.depth,
|
||||
depth_nonzero=live.depth_nonzero,
|
||||
dropped=row.dropped,
|
||||
)
|
||||
counts = provenance.counts
|
||||
counts["frame_record_bytes"] = counts.get("frame_record_bytes", 0) + len(record)
|
||||
if live.depth is not None:
|
||||
counts["depth_frames"] = counts.get("depth_frames", 0) + 1
|
||||
counts["depth_pixels"] = int(
|
||||
np.shape(live.depth)[-2]
|
||||
) * int(np.shape(live.depth)[-1])
|
||||
writer.add_frame(record, frame_id=row.frame_id, t_rel_s=row.t_rel_s)
|
||||
if self.on_progress is not None:
|
||||
self.on_progress(
|
||||
|
||||
@@ -0,0 +1,454 @@
|
||||
"""The depth channel in an analysis result: what it costs, and that it comes back.
|
||||
|
||||
What is being defended:
|
||||
|
||||
1. **The round trip is within the stated error, frame for frame, on real footage.** The
|
||||
chain is walked once more, independently of the pass, and every stored depth map is
|
||||
compared with the map the chain produced for the same frame id. The accepted error is
|
||||
half one quantisation step of that frame's own range, ``(max - min) / 510`` — below
|
||||
half a step of the 256-entry colour ramp the depth view paints with, so a
|
||||
reconstruction cannot be coloured differently from the original.
|
||||
2. **The size is stated and held.** 14 B per frame whose depth map is exactly all zeros
|
||||
(the frame header's statistics plus a 6-byte payload header) and the header plus a
|
||||
zlib'd uint8 plane for a frame that has one. The alternative encodings were measured
|
||||
and rejected; the numbers are in the module docstring of ``analysis.model``.
|
||||
3. **"Empty" and "absent" are different answers.** A frame whose depth map is all zeros
|
||||
still carries the channel and states its shape; a frame whose chain exposed no depth
|
||||
channel at all carries none. A view that draws the same blank picture for both is the
|
||||
one that misleads, so the reader distinguishes them and the statistics travel with
|
||||
the record.
|
||||
4. **A corrupt depth payload is refused, not decoded into a plausible array.**
|
||||
|
||||
The fixtures use real footage — the 33-frame box recording under ``dist/`` and the
|
||||
264-frame eight-times-slow clip under ``.scratch/sdk-stream-pipeline``, both driven
|
||||
through the **finger** chain. Where the SDK or the footage is absent the tests skip
|
||||
rather than fake a result: a synthetic array would defend neither the statistics
|
||||
observed on this footage nor the size they produce.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from orisys_app_shared.analysis.cache import ResultCache, scan_results
|
||||
from orisys_app_shared.analysis.model import (
|
||||
CODEC_EMPTY,
|
||||
CODEC_RAW,
|
||||
CODEC_ZLIB,
|
||||
DEPTH_HEADER,
|
||||
decode_depth,
|
||||
encode_depth,
|
||||
encode_frame,
|
||||
frame_payload_bytes,
|
||||
)
|
||||
from orisys_app_shared.analysis.reader import AnalysisReader
|
||||
from orisys_app_shared.analysis.service import (
|
||||
AnalysisService,
|
||||
ChainSpec,
|
||||
_OpenChain,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
STUDIO = REPO / "dist" / "OrisysStudio"
|
||||
RECORDING = (
|
||||
STUDIO
|
||||
/ "outputs"
|
||||
/ "recordings"
|
||||
/ "20260930T015028Z_session0002_40cb8e7441434feab48bb646afdba011"
|
||||
)
|
||||
FINGER_CONFIG = STUDIO / "config" / "sensors" / "finger" / "sensor.json"
|
||||
FINGER_EXPORT = STUDIO / "config" / "sensors" / "finger" / "assets" / "30fps-0703"
|
||||
#: The 264-frame clip Step 0 measured the pass's throughput on. It lives in another
|
||||
#: task's scratch, so it is optional: without it the clip half of the round trip is
|
||||
#: reported as skipped and the box recording still runs.
|
||||
CLIP = REPO / ".scratch" / "sdk-stream-pipeline" / "long_8x.mp4"
|
||||
CLIP_FRAMES = 264
|
||||
CLIP_FPS = 30.0
|
||||
|
||||
#: Measured on the two recordings, at the finger chain's own 300 x 280 depth grid.
|
||||
#: Empty frames and the worst non-empty frame are the two numbers the encoding is
|
||||
#: chosen on, so they are asserted rather than described.
|
||||
DEPTH_GRID = (300, 280)
|
||||
EMPTY_PAYLOAD_BYTES = DEPTH_HEADER.size
|
||||
EMPTY_RECORD_BYTES = frame_payload_bytes(330, with_validity=True, with_depth=True)
|
||||
FIXED_RECORD_BYTES = EMPTY_RECORD_BYTES - DEPTH_HEADER.size
|
||||
|
||||
#: The worst non-empty frame either recording produced: 336 of 84,000 pixels on the clip,
|
||||
#: whose quantised plane zlib's to 486 B. The bound below is what the format promises;
|
||||
#: the measured number is what the recordings delivered, and both are asserted.
|
||||
WORST_NONEMPTY_PIXELS = 336
|
||||
WORST_NONEMPTY_PAYLOAD_BYTES = 486
|
||||
|
||||
|
||||
def _require_inputs() -> None:
|
||||
missing = [
|
||||
str(path)
|
||||
for path in (RECORDING / "frames.csv", RECORDING / "raw_0001.mp4", FINGER_CONFIG)
|
||||
if not path.exists()
|
||||
]
|
||||
if missing:
|
||||
pytest.skip("analysis inputs are not present: " + ", ".join(missing))
|
||||
pytest.importorskip("orisys", reason="the SDK is needed to run the analysis pass")
|
||||
|
||||
|
||||
def _finger_chain() -> ChainSpec:
|
||||
return ChainSpec(
|
||||
sensor_type="finger", config_ref=FINGER_CONFIG, export_dir=FINGER_EXPORT
|
||||
)
|
||||
|
||||
|
||||
def _result_directory(root: Path) -> Path:
|
||||
directories = [entry.directory for entry in scan_results(root) if entry.complete]
|
||||
assert directories, f"no complete result under {root}"
|
||||
return directories[0]
|
||||
|
||||
|
||||
def _clip_recording(tmp_path: Path) -> Path:
|
||||
"""The clip as a one-segment recording: a link to the video plus a frames index.
|
||||
|
||||
A bare container has no ``frames.csv``, and frame identity comes from the
|
||||
recording's own counter (Part A, section 6). Writing the index here is what makes
|
||||
the clip addressable by the same code path as a real recording instead of by an
|
||||
index standing in for an id.
|
||||
"""
|
||||
directory = tmp_path / "clip-recording"
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
video = directory / "raw_0001.mp4"
|
||||
if not video.exists():
|
||||
try:
|
||||
os.link(CLIP, video)
|
||||
except OSError:
|
||||
video.write_bytes(CLIP.read_bytes())
|
||||
index = directory / "frames.csv"
|
||||
rows = ["frame_id,t_rel_s,segment,segment_frame,in_video,dropped"]
|
||||
rows.extend(
|
||||
f"{number},{number / CLIP_FPS:.6f},1,{number},1,0"
|
||||
for number in range(1, CLIP_FRAMES + 1)
|
||||
)
|
||||
index.write_text("\n".join(rows) + "\n", encoding="utf-8")
|
||||
return directory
|
||||
|
||||
|
||||
def _chain_depth(recording: Path, video: str = "raw_0001.mp4") -> list[np.ndarray]:
|
||||
"""Walk the chain once more and return the depth map it produced for every frame.
|
||||
|
||||
Deliberately a *separate* walk from the pass: comparing a stored result with the
|
||||
object the pass held would only prove the writer copied what it was given.
|
||||
"""
|
||||
from orisys_app_shared.acquisition.sdk_worker import process_chain
|
||||
|
||||
produced: list[np.ndarray] = []
|
||||
with tempfile.TemporaryDirectory() as working:
|
||||
with _OpenChain(_finger_chain(), recording / video, Path(working)) as opened:
|
||||
while True:
|
||||
ok, frame = opened.read()
|
||||
if not ok or frame is None:
|
||||
break
|
||||
if opened.feed(frame) is None:
|
||||
raise AssertionError("the chain could not unroll a frame")
|
||||
depth, error = process_chain(opened.sensor)
|
||||
assert not error, error
|
||||
assert depth is not None, "the finger chain exposed no depth channel"
|
||||
array = np.asarray(depth, dtype=np.float32)
|
||||
while array.ndim > 2 and array.shape[0] == 1:
|
||||
array = array[0]
|
||||
produced.append(np.ascontiguousarray(array))
|
||||
return produced
|
||||
|
||||
|
||||
def _tolerance(frame) -> float:
|
||||
"""Half one quantisation step of this frame's own stored range."""
|
||||
return (float(frame.depth_max) - float(frame.depth_min)) / 510.0
|
||||
|
||||
|
||||
def _compare(stored, produced: list[np.ndarray]) -> dict[str, int]:
|
||||
"""Compare every stored depth map with the chain's, frame for frame.
|
||||
|
||||
The order is the pass's own: the finger chain consumes no frame before the first
|
||||
readable one, so decoded frame *k* is stored frame *k*.
|
||||
"""
|
||||
frames = list(stored)
|
||||
assert len(frames) == len(produced), (
|
||||
f"{len(frames)} stored frames against {len(produced)} decoded"
|
||||
)
|
||||
stats = {"frames": len(frames), "empty": 0, "nonempty": 0, "worst_pixels": 0}
|
||||
for frame, original in zip(frames, produced):
|
||||
assert frame.has_depth, f"frame {frame.frame_id} lost its depth channel"
|
||||
assert frame.depth.shape == DEPTH_GRID, frame.depth.shape
|
||||
assert original.shape == frame.depth.shape
|
||||
if frame.depth_nonzero:
|
||||
stats["nonempty"] += 1
|
||||
stats["worst_pixels"] = max(stats["worst_pixels"], frame.depth_nonzero)
|
||||
else:
|
||||
stats["empty"] += 1
|
||||
assert not bool(np.any(original)), (
|
||||
f"frame {frame.frame_id} is stored as an empty depth map but the chain "
|
||||
"produced a non-zero one"
|
||||
)
|
||||
# The statistic the record states is the statistic of the map the chain made.
|
||||
assert frame.depth_nonzero == int(np.count_nonzero(original))
|
||||
assert float(frame.depth_max) == float(original.max())
|
||||
assert float(frame.depth_min) == float(original.min())
|
||||
error = float(np.max(np.abs(np.asarray(frame.depth) - original)))
|
||||
assert error <= _tolerance(frame), (
|
||||
f"frame {frame.frame_id} is {error} away from the chain's map, over the "
|
||||
f"{_tolerance(frame)} the encoding accepts"
|
||||
)
|
||||
# The reconstruction carries the same zero/non-zero verdict everywhere: every
|
||||
# observed non-zero value is far above the half step, so the count survives too.
|
||||
assert int(np.count_nonzero(frame.depth)) == frame.depth_nonzero
|
||||
return stats
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ the round trip
|
||||
@pytest.fixture(scope="module")
|
||||
def box_depth(tmp_path_factory):
|
||||
"""One real pass over the evidence recording, kept for the round trip and the size."""
|
||||
_require_inputs()
|
||||
root = tmp_path_factory.mktemp("analysis-depth-box")
|
||||
outcome = AnalysisService(
|
||||
RECORDING, chain=_finger_chain(), cache=ResultCache(root)
|
||||
).run()
|
||||
assert outcome.complete, outcome.reason
|
||||
return root
|
||||
|
||||
|
||||
def test_the_box_recording_round_trips_its_depth_within_the_stated_error(box_depth):
|
||||
produced = _chain_depth(RECORDING)
|
||||
reader = AnalysisReader(_result_directory(box_depth))
|
||||
try:
|
||||
stats = _compare(reader.frames(), produced)
|
||||
# The observed split, not a hope: 32 of the 33 frames carry a map that is
|
||||
# exactly all zeros, and the one that does not holds 76 pixels of 84,000.
|
||||
assert stats == {
|
||||
"frames": 33,
|
||||
"empty": 32,
|
||||
"nonempty": 1,
|
||||
"worst_pixels": 76,
|
||||
}
|
||||
finally:
|
||||
reader.close()
|
||||
|
||||
|
||||
def test_the_clip_round_trips_its_depth_within_the_stated_error(tmp_path):
|
||||
if not CLIP.is_file():
|
||||
pytest.skip(f"the clip is not present: {CLIP}")
|
||||
_require_inputs()
|
||||
recording = _clip_recording(tmp_path)
|
||||
root = tmp_path / "cache"
|
||||
outcome = AnalysisService(
|
||||
recording, chain=_finger_chain(), cache=ResultCache(root)
|
||||
).run()
|
||||
assert outcome.complete, outcome.reason
|
||||
|
||||
produced = _chain_depth(recording)
|
||||
reader = AnalysisReader(_result_directory(root))
|
||||
try:
|
||||
stats = _compare(reader.frames(), produced)
|
||||
assert stats == {
|
||||
"frames": CLIP_FRAMES,
|
||||
"empty": 241,
|
||||
"nonempty": 23,
|
||||
"worst_pixels": WORST_NONEMPTY_PIXELS,
|
||||
}
|
||||
finally:
|
||||
reader.close()
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------- the size
|
||||
def test_the_depth_cost_is_the_measured_encoding(box_depth):
|
||||
"""14 B per empty frame, the plane's own zlib'd bytes for a frame that has one.
|
||||
|
||||
The record without depth is 4,034 B at the finger grid; the depth channel adds the
|
||||
eight bytes of per-frame statistics the header now carries (``depth_min``,
|
||||
``depth_max``) and the six-byte payload header, and then whatever the plane costs.
|
||||
Ten minutes at 30 fps is 18,000 frames, so the empty case is 72,864,000 B = 69.5 MiB
|
||||
against 69.3 MiB before: the number is small because the *common* case is a map that
|
||||
is all zeros and a payload that is a header.
|
||||
"""
|
||||
assert DEPTH_HEADER.size == 6
|
||||
assert FIXED_RECORD_BYTES == 4042
|
||||
assert EMPTY_RECORD_BYTES == 4048
|
||||
|
||||
reader = AnalysisReader(_result_directory(box_depth))
|
||||
try:
|
||||
frames = list(reader.frames())
|
||||
emptied = [item for item in frames if item.depth_nonzero == 0]
|
||||
carrying = [item for item in frames if item.depth_nonzero]
|
||||
assert len(emptied) + len(carrying) == len(frames)
|
||||
assert {item.depth_stored_bytes for item in emptied} == {EMPTY_PAYLOAD_BYTES}
|
||||
for item in carrying:
|
||||
assert item.depth_stored_bytes == DEPTH_HEADER.size + len(
|
||||
_payload_after_header(item.depth, item.depth_min, item.depth_max)
|
||||
)
|
||||
assert reader.frame_record_bytes == sum(
|
||||
FIXED_RECORD_BYTES + item.depth_stored_bytes for item in frames
|
||||
)
|
||||
assert reader.bytes_per_frame <= 4096.0, "the stated per-frame bound"
|
||||
# Worst case, every frame the worst non-empty frame observed on the clip.
|
||||
worst = FIXED_RECORD_BYTES + DEPTH_HEADER.size + WORST_NONEMPTY_PAYLOAD_BYTES
|
||||
assert worst == 4534
|
||||
assert worst * 18_000 == 81_612_000, "81.6 MB per ten minutes, upper bound"
|
||||
# The rate the footage actually produced, over the 264-frame clip: 241 empty
|
||||
# frames and 23 at the worst non-empty size.
|
||||
typical = (241 * EMPTY_RECORD_BYTES + 23 * worst) / 264
|
||||
assert typical == pytest.approx(4090.34, abs=0.01)
|
||||
assert typical * 18_000 == pytest.approx(73_626_136, abs=1.0)
|
||||
assert typical * 18_000 < 73_700_000, "70.2 MB per ten minutes at the observed rate"
|
||||
assert typical < 4096.0
|
||||
finally:
|
||||
reader.close()
|
||||
|
||||
|
||||
def _payload_after_header(depth: np.ndarray, low: float, high: float) -> bytes:
|
||||
"""What ``encode_depth`` would store for a map, payload only. Shared with the codec."""
|
||||
return encode_depth(depth)[0][DEPTH_HEADER.size :]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- the codec itself
|
||||
def test_an_empty_map_costs_a_header_and_states_its_shape():
|
||||
depth = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
payload, nonzero, low, high = encode_depth(depth)
|
||||
assert (nonzero, low, high) == (0, 0.0, 0.0)
|
||||
assert payload == DEPTH_HEADER.pack(CODEC_EMPTY, 8, *DEPTH_GRID)
|
||||
assert len(payload) == EMPTY_PAYLOAD_BYTES
|
||||
|
||||
decoded = decode_depth(payload)
|
||||
assert decoded.is_empty and decoded.nonzero == 0
|
||||
assert (decoded.height, decoded.width) == DEPTH_GRID
|
||||
from orisys_app_shared.analysis.model import depth_values
|
||||
|
||||
restored = depth_values(decoded, low=low, high=high)
|
||||
assert restored.shape == DEPTH_GRID
|
||||
assert not bool(np.any(restored)), "an empty map comes back as zeros, not as NaNs"
|
||||
|
||||
|
||||
def test_a_single_plane_is_quantised_over_its_own_range_and_stays_within_half_a_step():
|
||||
rng = np.random.default_rng(20261004)
|
||||
depth = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
rows = rng.integers(0, DEPTH_GRID[0], 500)
|
||||
cols = rng.integers(0, DEPTH_GRID[1], 500)
|
||||
depth[rows, cols] = rng.random(500).astype(np.float32) * 0.05
|
||||
|
||||
payload, nonzero, low, high = encode_depth(depth)
|
||||
assert nonzero == int(np.count_nonzero(depth))
|
||||
assert (low, high) == (float(depth.min()), float(depth.max()))
|
||||
assert len(payload) < 4096, "a 500-pixel plane is not 84 kB of payload"
|
||||
|
||||
decoded = decode_depth(payload, low=low, high=high)
|
||||
assert decoded.codec in (CODEC_ZLIB, CODEC_RAW)
|
||||
from orisys_app_shared.analysis.model import depth_values
|
||||
|
||||
restored = depth_values(decoded, low=low, high=high)
|
||||
step = (high - low) / 255.0
|
||||
assert float(np.max(np.abs(restored - depth))) <= step / 2.0 + 1e-12
|
||||
# The count survives for every value the quantiser can actually represent: a value
|
||||
# above half a step lands on a code of at least one. A value *below* half a step can
|
||||
# round to zero, and that is the accepted error rather than a lost measurement — it
|
||||
# is smaller than the error the encoding already allows.
|
||||
assert int(np.count_nonzero(restored[depth > step / 2.0])) == int(
|
||||
np.count_nonzero(depth > step / 2.0)
|
||||
)
|
||||
assert not bool(np.any(restored[depth == 0.0]))
|
||||
|
||||
# The display paints this range onto a 256-entry ramp, so half a quantisation step
|
||||
# is half a ramp step: the reconstruction cannot change a pixel's colour.
|
||||
assert step / 2.0 <= (high - low) / 255.0
|
||||
|
||||
|
||||
def test_a_plane_that_does_not_compress_is_stored_raw_rather_than_larger():
|
||||
rng = np.random.default_rng(7)
|
||||
depth = rng.random(DEPTH_GRID).astype(np.float32)
|
||||
payload, nonzero, low, high = encode_depth(depth)
|
||||
decoded = decode_depth(payload, low=low, high=high)
|
||||
assert decoded.codec == CODEC_RAW
|
||||
assert decoded.stored_bytes == depth.size
|
||||
assert len(payload) == DEPTH_HEADER.size + depth.size
|
||||
|
||||
|
||||
def test_depth_statistics_are_per_frame_because_the_range_is():
|
||||
"""Two frames with different ranges each keep all 256 levels."""
|
||||
flat = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
flat[10, 10] = 0.001
|
||||
steep = np.zeros(DEPTH_GRID, dtype=np.float32)
|
||||
steep[10, 10] = 0.0
|
||||
steep[20, 20] = 1.0
|
||||
for frame in (flat, steep):
|
||||
payload, _, low, high = encode_depth(frame)
|
||||
decoded = decode_depth(payload, low=low, high=high)
|
||||
from orisys_app_shared.analysis.model import depth_values
|
||||
|
||||
restored = depth_values(decoded, low=low, high=high)
|
||||
assert float(np.max(np.abs(restored - frame))) <= (high - low) / 510.0 + 1e-12
|
||||
|
||||
|
||||
def test_a_corrupt_depth_payload_is_refused_rather_than_decoded():
|
||||
good, _, low, high = encode_depth(
|
||||
np.full(DEPTH_GRID, 0.25, dtype=np.float32)
|
||||
)
|
||||
# A plane that decodes to the wrong number of bytes for the shape it declares.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(good[:-1], low=low, high=high)
|
||||
# A shape no sensor has, refused before it becomes an allocation.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(DEPTH_HEADER.pack(CODEC_EMPTY, 8, 0xFFFF, 0xFFFF))
|
||||
# A bit depth this build does not store.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(DEPTH_HEADER.pack(CODEC_EMPTY, 16, *DEPTH_GRID))
|
||||
# And a header that is not even there.
|
||||
with pytest.raises(ValueError):
|
||||
decode_depth(b"")
|
||||
|
||||
|
||||
def test_a_frame_without_a_depth_channel_is_not_a_frame_with_an_empty_one():
|
||||
"""The two facts have to stay apart: only one of them can be drawn as a measurement."""
|
||||
blank = encode_frame(
|
||||
frame_id=1,
|
||||
t_rel_s=0.0,
|
||||
grid_id=0,
|
||||
dx=np.zeros(2, np.float32),
|
||||
dy=np.zeros(2, np.float32),
|
||||
magnitude=np.zeros(2, np.float32),
|
||||
active=np.zeros(2, bool),
|
||||
depth=np.zeros(DEPTH_GRID, np.float32),
|
||||
)
|
||||
absent = encode_frame(
|
||||
frame_id=2,
|
||||
t_rel_s=0.0,
|
||||
grid_id=0,
|
||||
dx=np.zeros(2, np.float32),
|
||||
dy=np.zeros(2, np.float32),
|
||||
magnitude=np.zeros(2, np.float32),
|
||||
active=np.zeros(2, bool),
|
||||
)
|
||||
assert len(blank) - len(absent) == EMPTY_PAYLOAD_BYTES
|
||||
from orisys_app_shared.analysis.model import decode_frame
|
||||
|
||||
empty_frame = decode_frame(blank)
|
||||
absent_frame = decode_frame(absent)
|
||||
assert empty_frame.depth is not None and empty_frame.depth.is_empty
|
||||
assert empty_frame.depth_map() is not None
|
||||
assert absent_frame.depth is None and absent_frame.depth_map() is None
|
||||
|
||||
|
||||
def test_the_provenance_counts_what_was_empty(box_depth):
|
||||
"""The operator's "there is no depth here" has to be answerable from the result."""
|
||||
reader = AnalysisReader(_result_directory(box_depth))
|
||||
try:
|
||||
counts = reader.counts
|
||||
assert counts["depth_frames"] == 33
|
||||
assert counts["depth_nonzero_frames"] == 1
|
||||
assert counts["depth_nonzero_max"] == 76
|
||||
assert counts["depth_pixels"] == DEPTH_GRID[0] * DEPTH_GRID[1]
|
||||
assert counts["frame_record_bytes"] == reader.frame_record_bytes
|
||||
provenance = json.loads(
|
||||
(_result_directory(box_depth) / "meta.json").read_text(encoding="utf-8")
|
||||
)["provenance"]
|
||||
assert provenance["sources"]["depth"] == "recomputed"
|
||||
finally:
|
||||
reader.close()
|
||||
@@ -38,7 +38,11 @@ from orisys_app_shared.analysis.cache import (
|
||||
evict,
|
||||
scan_results,
|
||||
)
|
||||
from orisys_app_shared.analysis.model import frame_payload_bytes, result_key
|
||||
from orisys_app_shared.analysis.model import (
|
||||
FRAME_HEADER,
|
||||
frame_payload_bytes,
|
||||
result_key,
|
||||
)
|
||||
from orisys_app_shared.analysis.reader import (
|
||||
AnalysisIncompleteError,
|
||||
AnalysisReader,
|
||||
@@ -477,24 +481,43 @@ def test_the_key_follows_the_inputs_that_change_the_result_and_no_others(finger_
|
||||
|
||||
# ------------------------------------------------------------------ size
|
||||
def test_the_stored_size_is_the_measured_encoding(box_pass, finger_pass):
|
||||
"""State the encoding, then hold it: 4,034 B per frame at the finger grid.
|
||||
"""State the encoding, then hold it: 4,048 B per frame at the finger grid.
|
||||
|
||||
Implicit candidate grid (a keyframe) + float32 ``dx``/``dy``/``magnitude`` +
|
||||
bit-packed ``active`` + a 24-byte header + the 8-byte validity digest:
|
||||
``24 + 330*12 + 42 + 8 = 4,034`` B. The alternative measured encodings are float16
|
||||
``dxy``+magnitude (2,022 B) and dropping magnitude (4,002 B with the grid, no
|
||||
header); float32 is chosen because the round trip has to be *exact*, and the
|
||||
magnitude is what serves a changed threshold.
|
||||
bit-packed ``active`` + a 32-byte header + the 8-byte validity digest + the 6-byte
|
||||
depth header: ``32 + 330*12 + 42 + 8 + 6 = 4,048`` B, of which the depth part is
|
||||
``14`` B over the 4,034 B this record cost before the depth channel was stored. A
|
||||
frame whose depth map is *not* all zeros adds its own zlib'd uint8 plane on top.
|
||||
The alternative measured encodings are float16 ``dxy``+magnitude (2,022 B) and
|
||||
dropping magnitude (4,002 B with the grid, no header); float32 is chosen because
|
||||
the round trip has to be *exact*, and the magnitude is what serves a changed
|
||||
threshold.
|
||||
"""
|
||||
assert frame_payload_bytes(330, with_validity=True) == 4034
|
||||
assert frame_payload_bytes(330, with_validity=False) == 4026
|
||||
assert frame_payload_bytes(400, with_validity=False) == 4874
|
||||
fixed = FRAME_HEADER.size + 330 * 12 + 42 + 8
|
||||
assert fixed == 4042
|
||||
assert frame_payload_bytes(330, with_validity=True) == 4042
|
||||
assert frame_payload_bytes(330, with_validity=True, with_depth=True) == 4048
|
||||
assert frame_payload_bytes(330, with_validity=False, with_depth=True) == 4040
|
||||
assert frame_payload_bytes(400, with_validity=False) == 4882
|
||||
assert frame_payload_bytes(400, with_validity=False, with_depth=True) == 4888
|
||||
|
||||
_, finger = finger_pass
|
||||
reader = AnalysisReader(finger.directory)
|
||||
try:
|
||||
assert reader.bytes_per_frame == 4034.0
|
||||
assert reader.frame_record_bytes == 33 * 4034
|
||||
# Every frame states the depth channel; the ones whose map is all zeros pay the
|
||||
# 6-byte header and nothing else.
|
||||
frames = list(reader.frames())
|
||||
emptied = [item for item in frames if item.has_depth and not item.depth_nonzero]
|
||||
carrying = [item for item in frames if item.depth_nonzero]
|
||||
assert len(emptied) + len(carrying) == len(frames)
|
||||
assert reader.frame_record_bytes == sum(
|
||||
fixed + item.depth_stored_bytes for item in frames
|
||||
)
|
||||
assert reader.frame_record_bytes == (
|
||||
len(emptied) * 4048
|
||||
+ len(carrying) * fixed
|
||||
+ sum(item.depth_stored_bytes for item in carrying)
|
||||
)
|
||||
assert reader.bytes_per_frame <= 4096, "the stated per-frame bound"
|
||||
total_per_frame = _directory_bytes(finger.directory) / len(reader)
|
||||
assert total_per_frame <= 4600, "keyframes and metadata included"
|
||||
@@ -505,8 +528,12 @@ def test_the_stored_size_is_the_measured_encoding(box_pass, finger_pass):
|
||||
box_reader = AnalysisReader(box.directory)
|
||||
try:
|
||||
# The box chain has no validity mask, so there is no digest and every frame
|
||||
# carries its own grid keyframe: 4,874 B of record per frame at 400 candidates.
|
||||
assert box_reader.frame_record_bytes == 26 * 4874
|
||||
# carries its own grid keyframe: 4,882 B of record per frame at 400 candidates,
|
||||
# plus whatever depth it carries.
|
||||
assert box_reader.frame_record_bytes == sum(
|
||||
32 + 400 * 12 + 50 + item.depth_stored_bytes
|
||||
for item in box_reader.frames()
|
||||
)
|
||||
assert box_reader.keyframe_counts == {"grid": 26}
|
||||
assert box_reader.mask() is None
|
||||
finally:
|
||||
@@ -608,7 +635,12 @@ def test_a_result_without_metadata_is_never_served(tmp_path):
|
||||
AnalysisReader(directory)
|
||||
|
||||
(directory / META_NAME).write_text(
|
||||
json.dumps({"format": analysis_model.ANALYSIS_FORMAT, "format_version": 1}),
|
||||
json.dumps(
|
||||
{
|
||||
"format": analysis_model.ANALYSIS_FORMAT,
|
||||
"format_version": analysis_model.ANALYSIS_FORMAT_VERSION,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
with pytest.raises(AnalysisIncompleteError):
|
||||
|
||||
Reference in New Issue
Block a user