orisys-tech/orisys-app
3
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Record the live box arrow grid so replay matches live paint

Browse Source 
Box analysis used the finger sample grid, so replay packed arrows into the contact zone. The pass now stores draw_arrows cells; stale box caches miss and re-run. Verified with shared and Studio tests, including a live-vs-replay canvas check.
This commit is contained in:
ykcheungab
2026-10-05 15:01:48 +08:00
parent 28e955a817
commit 3433eb0fec
10 changed files with 1614 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/studio/src/orisys_studio/replay_analysis.py
+107 -33
View File
@@ -24,15 +24,12 @@ ever speaks the second one.
and never a new pass. The colour range and the idle-dot switch are not even inputs
here: they live in the draw call, so changing one redraws and writes nothing.
The picture is rebuilt from the stored samples: ``dx``/``dy`` are the canvas
displacements the pass selected (uv pixels, already divided by the policy's arrow
scale), so the flow field is recovered by dividing them back out by
:data:`~orisys_app_shared.scene3d.visual_policy.DISPLACEMENT_PX_PER_FLOW` and writing
those values at the recorded candidate locations. The 2D canvas then runs its ordinary
path — the shared selector, the same idle dots, the same colour ramp — over that field.
This is exact, not approximate, and ``apps/studio/tests/test_replay_analysis.py``
asserts it against a pass over real footage: re-selecting the recovered field through
the display policy reproduces the stored sample set element for element.
The picture is rebuilt from the stored samples. For the shared painter, ``dx``/``dy``
are the drawn displacements, and the field is recovered by dividing them back out so
re-selecting it reproduces those offsets. A box result on the live painter's grid is
recovered from ``magnitude`` instead, because that painter colours by the displacement
before the length cap. ``apps/studio/tests/test_replay_analysis.py`` asserts the shared
path against a pass over real footage.
The 3D pane is handed the **stored** sample set directly (:attr:`AnalysedFrame.samples`)
— the object Part A documents as "what the canvas and the composer already draw" — so
@@ -49,6 +46,7 @@ from pathlib import Path
from typing import Any
import numpy as np
from orisys_app_shared.acquisition.arrows import uses_sdk_box_grid
from orisys_app_shared.acquisition.models import SdkPacket
from orisys_app_shared.analysis.reader import AnalysisReader
from orisys_app_shared.scene3d.visual_policy import (
@@ -67,11 +65,9 @@ __all__ = [
"policy_divergence",
]
#: The UV domain to rebuild the flow field in when the result carries no validity mask
#: keyframe (the box chain has none). Only the recorded candidate locations are ever
#: read back out of it, so its size is a canvas for the samples rather than a claim
#: about the device — but it has to hold every candidate, which
#: :meth:`AnalysisSession._recover_flow` checks.
#: Last-resort canvas when a result has no mask, no depth plane and no grid. The box
#: chain's samples live on ``resolution_stitch``, not on this raw-camera frame; using it
#: whenever the mask is absent paints that grid into one corner.
DEFAULT_DOMAIN_HW = (1080, 1920)
#: The policy keys that decide *where* the candidate locations are. A display setting
@@ -191,6 +187,7 @@ class AnalysisSession:
self._times = np.asarray(reader.times, dtype=np.float64)
self._mask: np.ndarray | None = None
self._mask_loaded = False
self._domain_hw: tuple[int, int] | None = None
self._cache_id: int | None = None
self._cache: AnalysedFrame | None = None
@@ -256,6 +253,42 @@ class AnalysisSession:
def threshold(self) -> float:
return float(self.display_policy.flow_on_px)
def paint_policy(self) -> VisualPolicy:
"""The policy this result's **recovered** flow field can be drawn with.
The two halves come from different places, on purpose. *Where* the candidates are is
the pass's own answer — the field is finite only at those cells, and
:meth:`serve_policy` refuses a display setting that moves the grid — so the grid
travels from the result's provenance. *Which* candidates are active is a display
question, and ``magnitude`` is stored precisely so it can be asked again later, so
the threshold travels from the display policy.
Without this, a display setting that moves the grid would be refused by
``serve_policy`` and then applied to the drawing anyway, re-selecting the recovered
field on a grid whose cells mostly hold ``NaN``: the canvas would go blank rather
than draw the result's own samples.
"""
stored = self.stored_policy()
display = self.display_policy
policy = VisualPolicy(
flow_on_px=display.flow_on_px,
flow_off_px=display.flow_off_px,
flow_grid_spacing=int(
stored.get("flow_grid_spacing", self.policy.flow_grid_spacing)
),
flow_arrow_scale=float(
stored.get("flow_arrow_scale", self.policy.flow_arrow_scale)
),
flow_edge_margin=int(
stored.get("flow_edge_margin", self.policy.flow_edge_margin)
),
flow_length_margin=float(
stored.get("flow_length_margin", self.policy.flow_length_margin)
),
max_arrows=int(stored.get("max_arrows", self.policy.max_arrows)),
)
return policy.normalised()
# ------------------------------------------------------------------ looking
def frame_id_at(self, t_rel_s: float) -> int | None:
"""The stored frame at or just before ``t_rel_s``, by *signal* time."""
@@ -351,10 +384,35 @@ class AnalysisSession:
return self._mask
def domain_hw(self) -> tuple[int, int]:
"""The image the stored samples were measured on.
A validity mask is that image. Without one (the box chain), the depth plane is
the stitch grid and the candidate coordinates live on it. The raw camera frame
is larger, and painting the stitch grid onto it leaves the arrows in a corner.
"""
mask = self.mask()
if mask is None:
return DEFAULT_DOMAIN_HW
return (int(mask.shape[0]), int(mask.shape[1]))
if mask is not None:
return (int(mask.shape[0]), int(mask.shape[1]))
if self._domain_hw is None:
self._domain_hw = self._unmasked_domain_hw()
return self._domain_hw
def _unmasked_domain_hw(self) -> tuple[int, int]:
height = width = 0
if self._frame_ids.size:
depth = getattr(self.reader.frame(int(self._frame_ids[0])), "depth", None)
if depth is not None and int(getattr(depth, "ndim", 0)) == 2 and depth.size:
height, width = int(depth.shape[0]), int(depth.shape[1])
for reference in self.reader.keyframes.values():
if getattr(reference, "kind", "") != "grid":
continue
rows, cols = self.reader.grid(int(reference.id))
if rows.size:
height = max(height, int(np.max(rows)) + 1)
width = max(width, int(np.max(cols)) + 1)
if height > 0 and width > 0:
return (height, width)
return DEFAULT_DOMAIN_HW
def _compose(self, payload: Any) -> AnalysedFrame:
mask = self.mask()
@@ -387,15 +445,17 @@ class AnalysisSession:
invalid, so no neighbouring cell can contribute a sample the pass did not
record.
``dx``/``dy`` are the **drawn** displacements, which the selector produced as
``DISPLACEMENT_PX_PER_FLOW * flow / flow_arrow_scale`` and then scaled by
``flow_length_margin`` when the arrow was longer than its cell. Recovering the
field is therefore a division by the arrow scale, the display constant *and*
that margin — measured, not assumed: dividing by the first two alone leaves every
arrow 20% short (the finger margin is 0.8), which changes 319 of 330 samples and
one frame's active set. With the margin included the selector's own
``np.where(length > spacing * margin, ...)`` branch is idempotent, so
re-selecting the recovered field reproduces the stored samples exactly.
``dx``/``dy`` are the **drawn** displacements: ``DISPLACEMENT_PX_PER_FLOW * flow
/ flow_arrow_scale``, then multiplied by ``flow_length_margin`` (and capped).
The shared painter re-selects this field, so the inverse divides by that
product and the cap stays idempotent: the stored offsets come back exactly.
At ``flow_arrow_scale == 1`` that divisor is ``DISPLACEMENT_PX_PER_FLOW`` times
the margin.
The box painter does not re-select. It colours by the displacement *before*
the cap, which ``magnitude`` still holds (``|DISPLACEMENT_PX_PER_FLOW * flow|``).
Direction comes from ``dx``/``dy``. Float32 magnitude is not exact enough to
reproduce a capped offset, which is why only the box grid uses it.
"""
rows = np.asarray(samples.rows, np.int64)
cols = np.asarray(samples.cols, np.int64)
@@ -405,18 +465,32 @@ class AnalysisSession:
"the result's candidate locations do not fit the domain it declares "
f"({height}x{width}); the mask keyframe and the grid keyframe disagree"
)
divisor = (
DISPLACEMENT_PX_PER_FLOW
* float(self.policy.flow_arrow_scale)
* float(self.policy.flow_length_margin)
)
dx = np.asarray(samples.dx, np.float64)
dy = np.asarray(samples.dy, np.float64)
flow = np.full((height, width, 2), np.nan, dtype=np.float64)
with np.errstate(invalid="ignore"):
# ``NaN`` is the point of the mask, and the selector's tip arithmetic on a
# discarded cell is defined to be discarded: say so here rather than let
# numpy warn about an invalid cast on every frame.
flow[rows, cols, 0] = np.asarray(samples.dx, np.float64) / divisor
flow[rows, cols, 1] = np.asarray(samples.dy, np.float64) / divisor
if uses_sdk_box_grid(self.policy):
magnitude = np.asarray(samples.magnitude, np.float64)
length = np.sqrt(dx * dx + dy * dy)
flow_mag = magnitude / DISPLACEMENT_PX_PER_FLOW
safe = np.maximum(length, 1e-12)
moving = length > 0.0
flow[rows, cols, 0] = np.where(moving, dx / safe * flow_mag, 0.0)
flow[rows, cols, 1] = np.where(moving, dy / safe * flow_mag, 0.0)
else:
scale = float(self.policy.flow_arrow_scale)
if scale <= 0.0:
scale = 1.0
divisor = (
DISPLACEMENT_PX_PER_FLOW
* float(self.policy.flow_length_margin)
/ scale
)
flow[rows, cols, 0] = dx / divisor
flow[rows, cols, 1] = dy / divisor
return flow
apps/studio/src/orisys_studio/replay_controller.py
+36 -3
View File
@@ -698,8 +698,16 @@ class ReplayController(QObject):
policy = policy_for_settings(settings)
if not session.serve_policy(policy):
moved = policy_divergence(session.stored_policy(), policy)
self._host.notify(t("playback.analysis_grid_mismatch", keys="·".join(moved)))
# A box result's grid is the live painter's, not a display preference.
# The display policy is the shared selector's, so the two always differ,
# and saying so would scold the operator for a grid they cannot change.
from orisys_app_shared.acquisition.arrows import uses_sdk_box_grid
if not uses_sdk_box_grid(session.stored_policy()):
moved = policy_divergence(session.stored_policy(), policy)
self._host.notify(
t("playback.analysis_grid_mismatch", keys="·".join(moved))
)
self._engine.redraw()
def on_analysis_requested(self) -> None:
@@ -810,6 +818,19 @@ class ReplayController(QObject):
return folder.parent
return None
@staticmethod
def _host_chain_is_box(service) -> bool:
chain = getattr(service, "chain", None)
return str(getattr(chain, "sensor_type", "") or "") == "box"
@staticmethod
def _stored_grid_is_sdk_box(reader) -> bool:
from orisys_app_shared.acquisition.arrows import uses_sdk_box_grid
provenance = getattr(reader, "provenance", None)
stored = dict(getattr(provenance, "policy", {}) or {})
return uses_sdk_box_grid(stored)
def _activate_stored_result(
self, recording: Path, *, notify_reuse: bool = False, reanalyse: bool = False
) -> bool:
@@ -844,6 +865,17 @@ class ReplayController(QObject):
reader = None
if reader is None:
return False
# A box result recorded on the shared selector's grid cannot be drawn by the
# live painter. Treat it as a miss so the pass runs again and records the
# painter's own cells. The key does not include the grid, so the new pass
# replaces this directory.
if self._host_chain_is_box(service) and not self._stored_grid_is_sdk_box(reader):
logger.info("stored box result uses a different sample grid; re-running")
try:
reader.close()
except Exception: # noqa: BLE001 - releasing is best effort
logger.debug("could not close the stale box result", exc_info=True)
return False
try:
session = AnalysisSession(
reader,
@@ -1153,7 +1185,8 @@ class ReplayController(QObject):
# drawable at all; the plan reads it to decide what to offer.
result_available=True,
# ...and the field it carries is *recovered*: finite only at the candidate
# cells the pass recorded, so it can only be re-selected on the recorded grid.
# cells the pass recorded. A finger result is re-selected on that grid; a
# box result recorded on the live painter's grid is filled and drawn by it.
# The display's own policy is the caller's for a live field, and wrong here.
arrow_policy=self._session_policy(),
)