refactor(studio-2d): 二维图像单一后端 —— 视图模式由帧内容决定,四条路径共用一个渲染入口

- What:
  1. 新增 `shared/.../acquisition/image_backend.py`(纯 NumPy,无 Qt):`ImageFrame`(底图 + flow/depth/
     valid_mask + source/frame_id/t_rel_s/stale)、`ImagePlan`、`plan_image()`、`render_image()`,以及模式
     常量(`arrows_pure`/`arrows_stitched`/`depth`/`recorded`)与来源常量(live/recompute/replay/preview)。
     `plan_image()` 分两个问题回答:**可选项**由来源决定(live/recompute = 三种派生视图;录像/摄像头预览
     = 只有 `recorded`),**这一帧能不能画**由 `missing` 说明(某帧缺 flow/depth 不会悄悄改掉操作者选的
     视图,而是在页脚写一行)。`render_image()` 是唯一渲染入口:内部复用 `draw_arrow_display`/
     `depth_map_to_bgr`/`flow_colormap`(公共签名不变),深度图**改用主题色带**(与面板色条同一张表——
     原来窗口没传 colormap,深度图走 OpenCV JET 而色条画品牌色带,图例与图片两张表)。
  2. `ui/arrow_panel.py`:新增 `bgr_to_qimage()`(全应用唯一一份 BGR→QImage,`app_window._bgr_to_qimage`
     成为它的别名、`replay.py` 的重复实现删除);新增 `set_plan(plan)`/`plan_modes()`——模式集来自 plan,
     `view_modes()` 在无 plan 时回退旧的预设式列表(`set_source_kind` 保留"命名/文案"职责,并支持
     `recompute` 状态与 `panels.recompute_pipeline`);页脚新增一行 `note_label` 说明 plan(如
     "录像只保存原始相机图像…"),箭头放大/有效区注解改由"本帧是否支持箭头视图"决定;
     `set_view_mode()` 与 `_populate_view_combo()` 按当前模式集校验(不再只认三个常量,否则 `recorded`
     会被强行改回 `arrows_pure`)。
  3. `app_window.py`:新增唯一入口 `_show_image_frame(ImageFrame)`(plan → 渲染 → 一个 QImage 给面板),
     **四条生产者全部改走它**:实时/重算(`_render_run_image`,来源按 `_recompute_running` 区分)、回放
     (`_render_replay_frame` 用 `replay.video_frame_bgr()`,不再自己转 QImage)、摄像头预览
     (`_on_preview_tick`)。`_render_run_image` 里那套按 `mode` 分支画箭头的代码整体删除。
  4. `replay.py`:新增 `video_frame_bgr()`(按光标缓存 BGR),`video_image()` 保留为兼容薄封装;删除
     本模块的 `_bgr_to_qimage`。
  5. locale:新增 `panels.replay_raw_note`(录像只有原始图像的说明 + 指路「由视频重算」)、
     `panels.frame_missing_note`(`{fields}` 参数化)、`panels.field_flow`/`panels.field_depth`、
     `panels.recompute_pipeline(_tip)`(中英各一套)。
- Why: 用户报的两个 bug 之一——「历史回放面板的 2 维图只能显示原始采集图像,没有进行任何处理」,并建议
  "每一个 2D 图的后端保持一致"。核对后确认根因是架构而非漏了一步渲染:二维有三个(实际四个)互不一致的
  生产者——实时/重算走 `draw_arrow_display`/`depth_map_to_bgr`,回放与预览把 BGR 直接交给面板;模式集与
  注解由**视图预设**决定(`arrow_panel.set_source_kind("replay")` 才切成 `recorded`),于是同一段视频在
  "回放"和"重算"下是两个成像栈,且 `_bgr_to_qimage` 有两份实现。用户已选定"统一后端 + 一键重算(不改记录
  格式)",本提交做前半段;"一键重算"的入口在下一提交。
- Verification:
  · 新增 `shared/tests/test_image_backend.py` 15 项:plan 的三类来源(live/recompute 三种派生视图、
  replay/preview 只有 recorded)、`missing`/`note_key`/`note_fields`、模式回退、**与旧渲染逐像素一致**
  (arrows_pure/arrows_stitched 对 `draw_arrow_display`、depth 对 `depth_map_to_bgr` 带主题色带)、
  原始帧不做缩放/不替图、缺 depth → None、缺 flow → 底图、灰度帧只在这里加宽一次。
  · `apps/studio/tests/test_ui_smoke.py` +2:同一段视频在回放路径只出 `recorded` 且有说明行、在重算/实时
  路径出三种派生视图且都经过同一个 `render_image`(用 spy 断言来源与模式)、面板 `note_label` 不是裸键、
  combo 项数与模式集一致;以及"全应用只有一处 BGR→QImage"的结构断言。
  · 既有 `test_depth_view_reports_missing_data_and_fixes_its_scale`(缺深度必须报"本帧没有深度数据"而不是
  回退显示普通形变图)继续通过——新的"缺字段说明 + 不回退替图"语义与它一致。
  · 全量:apps/studio 236 项 0 失败 14 跳过;shared 331 项 0 失败 1 跳过;ui_check 587 项 0 失败;
  locale_check 0 缺失。
- Not in this change: 「一键重算」按钮与"实时界面禁止回放入口"(7.2/7.3);深度图改用主题色带改变了 depth
  视图的颜色(这是修正:色条一直画的就是主题色带);三维(scene3d)未动;录像格式未动(仍只存原始相机图像)。
This commit is contained in:
ykcheungab
2026-09-14 21:24:28 +08:00
parent bf643de93d
commit 8e6f9658b4
8 changed files with 732 additions and 90 deletions
@@ -0,0 +1,220 @@
"""One place that turns a frame into the picture the 2D panel shows.
The application used to have four producers of that picture and two different imaging
stacks: live and recomputed frames went through ``draw_arrow_display``/
``depth_map_to_bgr`` in the window, the preview and the replay path handed a raw BGR
frame straight to the panel, and the panel decided which view modes existed from the
*view preset* (``set_source_kind("replay")``) rather than from what the frame actually
carried. The same video therefore behaved differently in "replay" and "recompute", and
a mode that needed data nobody had could still be offered.
This module is the single backend. Two rules make the difference:
* **the frame says what it has** (:class:`ImageFrame`), and the plan is derived from
that — not from which view is on screen. A frame with no flow has no arrow modes; a
frame with no depth has no depth mode; a frame with neither is a raw picture;
* **rendering is one function** (:func:`render_image`), so live, recompute, replay and
preview cannot drift apart. A mode the frame cannot support falls back to the plan's
default instead of drawing something plausible.
NumPy only: the Qt conversion belongs to the UI layer (``ui/arrow_panel.bgr_to_qimage``).
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Optional, Sequence
import numpy as np
from orisys_app_shared.acquisition.arrows import (
depth_map_to_bgr,
draw_arrow_display,
flow_colormap,
)
from orisys_app_shared.acquisition.models import ensure_bgr
#: The view modes, in the order a panel should offer them. ``recorded`` is not a
#: choice: it is what is left when the frame carries no derived field at all.
ARROW_PURE = "arrows_pure"
ARROW_STITCHED = "arrows_stitched"
DEPTH = "depth"
RECORDED = "recorded"
DERIVED_MODES = (ARROW_PURE, ARROW_STITCHED, DEPTH)
ALL_MODES = (*DERIVED_MODES, RECORDED)
#: Where a frame came from. Only used for wording and for the recording; the modes
#: come from the frame's own fields.
SOURCE_LIVE = "live"
SOURCE_RECOMPUTE = "recompute"
SOURCE_REPLAY = "replay"
SOURCE_PREVIEW = "preview"
#: Shown when the only thing a frame can be is a raw picture.
RAW_NOTE_KEY = "panels.replay_raw_note"
#: Shown when a frame of a *derived* source is missing a field (a dropped frame, a
#: failed tracker): the mode stays selectable — that is the operator's choice — and the
#: line says which field is absent instead of the view looking broken.
MISSING_NOTE_KEY = "panels.frame_missing_note"
#: Sources whose frames carry derived fields.
DERIVED_SOURCES = (SOURCE_LIVE, SOURCE_RECOMPUTE)
@dataclass(frozen=True)
class ImageFrame:
"""A picture to draw, plus whatever derived fields came with it.
``image_bgr`` is the *base* picture (the processed/unrolled frame for a live or
recomputed source, the recorded or camera frame otherwise). ``flow``/``depth``/
``valid_mask`` are the same arrays the rest of the pipeline uses; ``None`` means the
frame does not have them, and the plan will say so rather than offer a mode that
would have to invent them.
"""
image_bgr: Optional[np.ndarray] = None
flow: Optional[np.ndarray] = None
depth: Optional[np.ndarray] = None
valid_mask: Optional[np.ndarray] = None
source: str = SOURCE_LIVE
frame_id: int = 0
t_rel_s: float = 0.0
#: True when the producer knows this frame's measurement is not current (the
#: coordinator's "stale" state); the panel can say so under the picture.
stale: bool = False
def has_flow(self) -> bool:
return self.flow is not None
def has_depth(self) -> bool:
return self.depth is not None
@dataclass(frozen=True)
class ImagePlan:
"""What this frame can be shown as, and what is missing from it."""
available_modes: tuple[str, ...]
default_mode: str
missing: tuple[str, ...]
source: str
frame_id: int
t_rel_s: float
stale: bool
#: An i18n key for the one line that explains the plan, or ``""``. When
#: :attr:`note_fields` is non-empty the line takes it as ``fields``.
note_key: str = ""
note_fields: tuple[str, ...] = ()
def accepts(self, mode: str) -> bool:
return str(mode) in self.available_modes
def resolve(self, mode: str) -> str:
"""The mode to draw: the request, or the default when it cannot be drawn."""
wanted = str(mode or "")
return wanted if self.accepts(wanted) else self.default_mode
def needs(self, mode: str) -> Optional[str]:
"""The field a mode needs that this frame does not have, if any."""
wanted = str(mode or "")
if wanted in (ARROW_PURE, ARROW_STITCHED) and not self.has_flow():
return "flow"
if wanted == DEPTH and not self.has_depth():
return "depth"
return None
def has_flow(self) -> bool:
return "flow" not in self.missing
def has_depth(self) -> bool:
return "depth" not in self.missing
def plan_image(frame: ImageFrame) -> ImagePlan:
"""Derive the modes from the frame's *source* and its own fields.
Two different questions, deliberately answered in two different places:
* **which views may the operator choose?** For a source that carries derived fields
(live, recompute) all of them — a single dropped frame must not silently switch
the operator's view. For a source that never does (a recording, a camera preview)
exactly one, because offering the others would be offering a lie;
* **what can this frame actually be drawn as?** ``missing`` says which field is
absent, and the panel states it in one line.
"""
derived = str(frame.source or SOURCE_LIVE) in DERIVED_SOURCES
missing: list[str] = []
if not frame.has_flow():
missing.append("flow")
if not frame.has_depth():
missing.append("depth")
if not derived:
return ImagePlan(
available_modes=(RECORDED,),
default_mode=RECORDED,
missing=("flow", "depth"),
source=str(frame.source or SOURCE_LIVE),
frame_id=int(frame.frame_id),
t_rel_s=float(frame.t_rel_s),
stale=bool(frame.stale),
note_key=RAW_NOTE_KEY,
)
default = ARROW_PURE if frame.has_flow() else (DEPTH if frame.has_depth() else ARROW_PURE)
return ImagePlan(
available_modes=DERIVED_MODES,
default_mode=default,
missing=tuple(missing),
source=str(frame.source or SOURCE_LIVE),
frame_id=int(frame.frame_id),
t_rel_s=float(frame.t_rel_s),
stale=bool(frame.stale),
note_key=MISSING_NOTE_KEY if missing else "",
note_fields=tuple(missing),
)
def render_image(
frame: ImageFrame,
plan: Optional[ImagePlan] = None,
*,
mode: str = "",
theme: str = "dark",
draw_style: str = "box",
depth_range: Optional[tuple[float, float]] = None,
canvas_bgr: Optional[Sequence[int]] = None,
) -> Optional[np.ndarray]:
"""Draw ``frame`` in ``mode``; ``None`` when there is nothing to draw.
The mode is resolved through the plan first, so a caller that asks for a view the
frame cannot support gets the plan's default rather than a picture assembled from
an array that is not there.
"""
plan = plan or plan_image(frame)
wanted = plan.resolve(mode)
if wanted == DEPTH:
# Never substitute another picture for a missing depth map: that made "no
# depth" look like a depth reading. The panel says so instead.
return depth_map_to_bgr(
frame.depth,
flow_colormap(theme),
value_range=depth_range,
invalid_mask=frame.valid_mask,
)
if frame.image_bgr is None:
return None
if wanted in (ARROW_PURE, ARROW_STITCHED):
if frame.flow is None:
# The base picture with no arrows, and the plan's ``missing`` says why.
return ensure_bgr(np.asarray(frame.image_bgr))
return draw_arrow_display(
frame.image_bgr,
frame.flow,
background="captured" if wanted == ARROW_STITCHED else "pure_color",
canvas_bgr=canvas_bgr,
draw_style=draw_style,
valid_mask=frame.valid_mask,
colormap=flow_colormap(theme),
)
# ``recorded``: the frame is the picture, with nothing drawn over it.
return ensure_bgr(np.asarray(frame.image_bgr))
+192
View File
@@ -0,0 +1,192 @@
"""Acceptance for the single 2D image backend.
The point of this module is that live, recomputed, replayed and previewed frames go
through *one* renderer and that the view modes come from what the frame carries. These
tests pin both halves: a plan derived from fields (not from a view preset), and
pixel-for-pixel agreement with the primitives the application used before.
"""
from __future__ import annotations
import numpy as np
import pytest
from orisys_app_shared.acquisition import image_backend as backend
from orisys_app_shared.acquisition.arrows import (
depth_map_to_bgr,
draw_arrow_display,
flow_colormap,
)
def _image(h: int = 12, w: int = 16) -> np.ndarray:
return np.arange(h * w * 3, dtype=np.uint8).reshape(h, w, 3)
def _flow(h: int = 12, w: int = 16) -> np.ndarray:
flow = np.zeros((h, w, 2), dtype=np.float32)
flow[..., 0] = 3.0
flow[..., 1] = -1.5
return flow
# ------------------------------------------------------------------------- plans
def test_a_live_frame_offers_the_derived_modes_and_needs_no_note():
plan = backend.plan_image(
backend.ImageFrame(_image(), flow=_flow(), depth=np.ones((12, 16)))
)
assert plan.available_modes == ("arrows_pure", "arrows_stitched", "depth")
assert plan.default_mode == "arrows_pure"
assert plan.missing == ()
assert plan.note_key == ""
def test_a_recorded_frame_is_raw_and_says_so():
"""No flow and no depth: the frame *is* the picture, and the note explains it."""
plan = backend.plan_image(backend.ImageFrame(_image(), source=backend.SOURCE_REPLAY))
assert plan.available_modes == ("recorded",)
assert plan.default_mode == "recorded"
assert plan.missing == ("flow", "depth")
assert plan.note_key == backend.RAW_NOTE_KEY
assert plan.source == "replay"
def test_a_derived_source_keeps_every_mode_and_reports_what_is_missing():
"""A dropped frame must not silently switch the operator's view.
Live and recomputed frames may be missing a field for one frame; the mode stays
selectable and ``missing`` says which field is absent, so the panel can state it
instead of the view looking broken.
"""
flow_only = backend.plan_image(backend.ImageFrame(_image(), flow=_flow()))
assert flow_only.available_modes == backend.DERIVED_MODES
assert flow_only.missing == ("depth",)
assert flow_only.default_mode == "arrows_pure"
assert flow_only.note_key == backend.MISSING_NOTE_KEY
assert flow_only.note_fields == ("depth",)
assert flow_only.needs("depth") == "depth"
assert flow_only.needs("arrows_pure") is None
depth_only = backend.plan_image(backend.ImageFrame(_image(), depth=np.ones((12, 16))))
assert depth_only.available_modes == backend.DERIVED_MODES
assert depth_only.missing == ("flow",)
assert depth_only.default_mode == "depth"
assert depth_only.needs("arrows_pure") == "flow"
complete = backend.plan_image(
backend.ImageFrame(_image(), flow=_flow(), depth=np.ones((12, 16)))
)
assert complete.missing == ()
assert complete.note_key == ""
def test_a_raw_source_cannot_offer_a_derived_view_at_all():
"""A recording and a camera preview have no derived field, ever."""
for source in (backend.SOURCE_REPLAY, backend.SOURCE_PREVIEW):
plan = backend.plan_image(backend.ImageFrame(_image(), source=source))
assert plan.available_modes == ("recorded",)
assert plan.note_key == backend.RAW_NOTE_KEY
assert plan.note_fields == ()
def test_the_plan_carries_the_frame_identity_for_the_panel():
plan = backend.plan_image(
backend.ImageFrame(
_image(), source=backend.SOURCE_RECOMPUTE, frame_id=42, t_rel_s=1.25, stale=True
)
)
assert (plan.source, plan.frame_id, plan.t_rel_s, plan.stale) == ("recompute", 42, 1.25, True)
def test_a_mode_the_plan_does_not_offer_falls_back_to_the_default():
"""Never draw a view the source cannot have at all."""
recorded = backend.plan_image(
backend.ImageFrame(_image(), source=backend.SOURCE_REPLAY)
)
assert recorded.resolve("arrows_pure") == "recorded"
assert recorded.resolve("nonsense") == "recorded"
live = backend.plan_image(backend.ImageFrame(_image(), flow=_flow()))
assert live.resolve("recorded") == "arrows_pure", "a derived source has no raw mode"
assert live.resolve("arrows_stitched") == "arrows_stitched"
# --------------------------------------------------------------- one renderer
def test_the_backend_renders_what_the_primitives_rendered_before():
"""Live went through these two calls; the backend must not change a pixel."""
image = _image()
flow = _flow()
mask = np.ones((12, 16), dtype=bool)
mask[0, 0] = False
frame = backend.ImageFrame(image, flow=flow, valid_mask=mask)
plan = backend.plan_image(frame)
for mode in ("arrows_pure", "arrows_stitched"):
expected = draw_arrow_display(
image,
flow,
background="captured" if mode == "arrows_stitched" else "pure_color",
draw_style="finger",
valid_mask=mask,
)
got = backend.render_image(frame, plan, mode=mode, draw_style="finger")
assert np.array_equal(got, expected), mode
depth = np.linspace(0.0, 1.0, 12 * 16, dtype=np.float32).reshape(12, 16)
depth_frame = backend.ImageFrame(image, depth=depth, valid_mask=mask)
# The depth picture uses the *theme's* ramp — the same table the panel's colour bar
# paints. The window used to leave this argument out, so the map was drawn with
# OpenCV's JET while the legend under it showed the brand ramp: two colour tables
# for one picture, and the legend claiming they were the same.
expected_depth = depth_map_to_bgr(
depth, flow_colormap("dark"), value_range=(0.0, 1.0), invalid_mask=mask
)
got_depth = backend.render_image(
depth_frame, mode="depth", depth_range=(0.0, 1.0)
)
assert np.array_equal(got_depth, expected_depth)
def test_a_recorded_frame_is_passed_through_and_never_resized_to_fit():
image = _image(9, 7)
got = backend.render_image(
backend.ImageFrame(image, source=backend.SOURCE_REPLAY), mode="recorded"
)
assert got is not None and got.shape == image.shape
assert np.array_equal(got, image)
def test_nothing_to_draw_is_none_not_a_blank_picture():
assert backend.render_image(backend.ImageFrame(), mode="recorded") is None
assert backend.render_image(
backend.ImageFrame(None, flow=_flow()), mode="arrows_pure"
) is None
assert backend.render_image(backend.ImageFrame(None, depth=None), mode="depth") is None
def test_a_grey_base_frame_is_widened_once_by_the_backend():
"""A 2-D frame must reach the shader as BGR, and only one place may do it."""
grey = np.zeros((8, 6), dtype=np.uint8)
got = backend.render_image(backend.ImageFrame(grey), mode="recorded")
assert got is not None and got.shape == (8, 6, 3)
assert np.array_equal(got, np.stack([grey] * 3, axis=-1))
@pytest.mark.parametrize("mode", list(backend.ALL_MODES))
def test_every_offered_mode_renders_something_or_says_it_cannot(mode):
"""A mode the plan offers must never come back empty."""
frame = backend.ImageFrame(_image(), flow=_flow(), depth=np.ones((12, 16)))
plan = backend.plan_image(frame)
if not plan.accepts(mode):
assert plan.resolve(mode) in plan.available_modes
return
assert backend.render_image(frame, plan, mode=mode) is not None
def test_a_missing_field_is_reported_not_invented():
"""Depth with no depth says so; arrows with no flow show the base picture."""
no_depth = backend.ImageFrame(_image(), flow=_flow())
assert backend.render_image(no_depth, mode="depth") is None
no_flow = backend.ImageFrame(_image(), depth=np.ones((12, 16)))
base = backend.render_image(no_flow, mode="arrows_pure")
assert base is not None
assert np.array_equal(base, _image()), "the base picture must not be redrawn"