Merge pull request 'dd02 setting tune + change Class name from UVUnrollFrameProcessor to FrameUnwarper' (#5) from feature/finger_docs into dev

Reviewed-on: http://100.106.67.31:3000/orisys-tech/orisys-sdk/pulls/5
2026-07-14 20:09:53 +08:00
parent fe6c880473 9ee6ab9470
commit 3d1509df9d
14 changed files with 275 additions and 94 deletions
@@ -2,10 +2,10 @@


 a = Analysis(
-    ['D:\\repo\\orisyssdk_git\\examples\\finger_pipeline_3dview.py'],
+    ['D:\\repo\\orisys-sdk\\examples\\finger_pipeline_3dview.py'],
    pathex=[],
    binaries=[],
-    datas=[('D:\\repo\\orisyssdk_git\\config', 'config'), ('D:\\repo\\orisyssdk_git\\assets', 'assets')],
+    datas=[('D:\\repo\\orisys-sdk\\config', 'config'), ('D:\\repo\\orisys-sdk\\assets', 'assets')],
    hiddenimports=['PyQt5', 'pyqtgraph', 'open3d', 'cv2', 'orisys', 'gui', 'pipeline', 'viz', 'gui.app_paths', 'plotly', '_plotly_utils', 'open3d.cpu.pybind'],
    hookspath=[],
    hooksconfig={},
@@ -54,15 +54,6 @@ import orisys
 print(orisys.__version__)
 ```

-5 安装opencl（实验）:
-
-在不带nvidia显卡的设备，可选择通过opencl进行gpu 计算加速，需要确保opencl的库或驱动已经安装，然后在conda环境中安装依赖：
-
-``` bash
-pip install pyopencl
-conda install conda-forge::pyvkfft 
-```
-
 ***

 ## `Sensor` 类别
@@ -245,32 +236,32 @@ flow = sensor.read_info(sensor.info.VRAW)
 ***
 ***

-## `orisys.finger` 指纹 UV pipeline
+## `orisys.finger` 手指 UV pipeline

-`examples/finger_pipeline.py` 展示了一个更适合指纹/单相机场景的高层 API：
+`examples/finger_pipeline.py` 展示了一个更适合手指/单相机场景的高层 API：

 1. 使用 `FingerRuntime.open()` 打开视频源、加载 Blender 导出的资产，并构建 UV 展开映射。
 2. 用 `grab_and_process()` 或 `read_source() + process_frame()` 获取一帧展开后的触觉图像。
 3. 通过 `runtime.tactile`（`TactileProcessor`）调用 `compute_deformation()`、`compute_contact()`、`read_info()`。
 4. 退出前调用 `runtime.close()` 释放视频源和内部 `Sensor` 资源。

-相比直接使用 `Sensor`，`FingerRuntime` 已封装了视频读取、相机尺寸设置、UV 展开和有效区域掩码传递，更适合作为指纹 2D/3D 示例与上层应用的入口。
+相比直接使用 `Sensor`，`FingerRuntime` 已封装了视频读取、相机尺寸设置、UV 展开和有效区域掩码传递，更适合作为手指 2D/3D 示例与上层应用的入口。

 ### 1. `FingerRuntime.open` 方法

 #### 描述

-创建一个可复用的指纹 UV runtime。该方法会完成以下初始化：
+创建一个可复用的手指 UV runtime。该方法会完成以下初始化：

 - 解析 `export_dir` 下的 `mesh.npz`、`camera_intrinsics.npz`、`world_to_cam.npy`
 - 根据配置和标定信息构建/加载 `unroll_map_uv.npz`
 - 打开相机或视频文件
 - 创建内部 `Sensor("external", ...)`
- 创建 `UVUnrollFrameProcessor` 与 `TactileProcessor`
+- 创建 `FrameUnwarper` 与 `TactileProcessor`

 #### 输入参数

-* **export_dir** (`str`，可选): 指纹导出资产目录，默认值来自 `orisys.finger.DEFAULT_FINGER_EXPORT_DIR`。目录下通常需要包含 `mesh.npz`、`camera_intrinsics.npz`、`world_to_cam.npy`。
+* **export_dir** (`str`，可选): 手指导出资产目录，默认值来自 `orisys.finger.DEFAULT_FINGER_EXPORT_DIR`。目录下通常需要包含 `mesh.npz`、`camera_intrinsics.npz`、`world_to_cam.npy`。
 * **config** (`str`，可选): Orisys 配置文件路径，默认 `./config/dd02-ov.json`。
 * **map_cache** (`str`，可选): UV 展开缓存文件路径，默认值来自 `orisys.finger.DEFAULT_FINGER_MAP_CACHE`。
 * **source_scale** (`float`，可选): 输入相机分辨率缩放比例；未提供时根据配置自动推断。
@@ -402,7 +393,7 @@ runtime = FingerRuntime.open(

 除 `push_frame()` / `warmup()` 之外，`TactileProcessor` 的 `compute_deformation()`、`compute_contact()`、`read_info()`、`reset()` 调用方式与 `Sensor` 基本一致，因此可直接复用前文 `Sensor` 章节中的参数说明。

-### 10. 最小指纹 pipeline 示例
+### 10. 最小手指 pipeline 示例

 下面的流程与 `examples/finger_pipeline.py` 一致：

@@ -1,16 +1,16 @@
 叠动科技软件使用手册  
 版本： 0.4.1  
-更新日期： 2026年4月16日
+更新日期： 2026年7月13日

 # 简介
 本文件介绍如何上手和运行叠动视触觉传感器配套的开发套件。本套件是配合叠动视触觉传感器进行图像处理，计算形变、力以及接触信息。它可以从摄像头或视频文件中读取图像，处理后打印和展示处理后信息。  
 支持操作系统：Windows、Ubuntu  
-建议Python 版本: 3.10
+建议Python 版本: 3.10, 3.11, 3.12

 # 基础环境设置 
 ## 安装 Python 与 Anaconda
 1.	访问 [Anaconda 官网](https://www.anaconda.com/download)下载适合您操作系统的安装包（Windows、Linux）。
-2.	按照提示完成安装。（Widnows：建议在安装过程中勾选“将 Anaconda 添加到系统环境变量”选项，方便后续使用）。
+2.	按照提示完成安装。
 3.	安装完成后，打开命令行终端（Windows 用户可在”Anaconda Navigator”打开“Anaconda Prompt”），输入 `conda info` ，若能显示 conda 的信息，则说明安装成功。

 ## 创建conda虚拟环境
@@ -115,8 +115,8 @@ print(orisys.__version__)
 套件里面包含了多个例子，建议都在仓库根目录运行（即 `config/`、`examples/`、`assets/` 同级目录）。常见入口分为两类：

 1. `examples/basic_pipeline.py`：复眼/拼接传感器基础 OpenCV 示例
-2. `examples/finger_pipeline.py`：指纹传感器 UV 展开最小流程示例
-3. `examples/finger_pipeline_3dview.py`：指纹传感器 Qt 3D 可视化界面
+2. `examples/finger_pipeline.py`：手指传感器2维显示最小流程示例
+3. `examples/finger_pipeline_3dview.py`：手指传感器 Qt 3D 可视化界面

 下面先介绍最基础的 `basic_pipeline.py` 运行方式。

@@ -140,9 +140,9 @@ print(orisys.__version__)
    ```
 5.	程序会运行，并自动打开可视化窗口和打印资料，在窗口上按键盘”q”键可以停止运行，按”r”键可以重设数据

-### 指纹传感器 UV 示例
+### 手指传感器 UV 示例

-如果您使用的是指纹/单相机 UV 展开流程，可运行：
+如果您使用的是手指/单相机 UV 展开流程，可运行：

 ``` bash
 python examples/finger_pipeline.py --camera_id 0 --config ./config/dd02-ov.json
@@ -164,30 +164,30 @@ python examples/finger_pipeline.py --video ./data/sample.mp4 --config ./config/d
 常用参数：

 - `--verbose`：输出更详细的初始化日志
- `--profile`：打印每个阶段的耗时统计
- `--profile-every N`：每 N 帧输出一次平均耗时

 例如：

 ``` bash
-python examples/finger_pipeline.py --camera_id 0 --config ./config/dd02-ov.json --profile --profile-every 30
+python examples/finger_pipeline.py --camera_id 0 --config ./config/dd02-ov.json
 ```

-### 指纹 3D 可视化界面
-
-如果已经安装 GUI 依赖，还可以运行：
+### 手指 3D 可视化界面

+如果要使用 GUI ,安装GUI依赖：
+``` bash
+pip install open3d
+pip install pyqt5 pyqtgraph
+```
+然后运行：
 ``` bash
-pip install -e ".[gui]"
 python examples/finger_pipeline_3dview.py --video 0 --config ./config/dd02-ov.json
 ```

 该界面会提供：

- 指纹 UV 展开图
+- 手指平面化光流箭头显示
 - 力曲线
- 3D mesh 视图
- 录制与重置等交互控件
+- 3D 视图

 ## 开发套件使用方法
 参考开发套件附带的开发文档。
@@ -3,7 +3,7 @@
 Minimum UV-unroll finger tactile pipeline.

 Offline: load Blender export -> build_or_load_domain -> unroll_map_uv.npz
-Runtime:  camera frame -> UVUnrollFrameProcessor -> TactileProcessor -> forces/contact
+Runtime:  camera frame -> FrameUnwarper -> TactileProcessor -> forces/contact

 For the full demo (UV debug panels, dashboard, benchmarks) see:
  tools/3d_patch_generation_blender/unroll_realtime.py
@@ -9,4 +9,5 @@ scipy==1.13
 setuptools>=45.2.0
 av
 numba
-
+pyqt5
+pyqtgraph
@@ -6,7 +6,7 @@ import cv2
 import numpy as np

 from ..configs import load_config
-from ..frame_processors import UVUnrollFrameProcessor
+from ..frame_processors import FrameUnwarper
 from ..sensor import Sensor
 from ..unroll import (
    build_or_load_domain,
@@ -208,7 +208,7 @@ class FingerRuntime:
        geometry: FingerGeometry,
        domain,
        tactile: TactileProcessor,
-        frame_processor: UVUnrollFrameProcessor,
+        frame_processor: FrameUnwarper,
        capture,
    ):
        self.geometry = geometry
@@ -268,7 +268,7 @@ class FingerRuntime:
            raise RuntimeError("Cannot open camera/video source")

        sensor = Sensor("external", config_name=config, verbose=verbose, mirror=mirror)
-        frame_processor = UVUnrollFrameProcessor(
+        frame_processor = FrameUnwarper(
            domain.remap,
            camera_matrix=geometry.source_K if geometry.dist_coeffs is not None else None,
            dist_coeffs=geometry.dist_coeffs,
@@ -30,7 +30,7 @@ class SingleCameraFrameProcessor:
        return frame


-class UVUnrollFrameProcessor:
+class FrameUnwarper:
    """Remap a raw camera frame into a flattened UV-domain BGR image."""

    def __init__(
@@ -230,7 +230,7 @@ def create_frame_processor(sensor_type, stitcher=None, uv_remap=None, **uv_kwarg
    if sensor_type in ("uv_unroll", "unroll"):
        if uv_remap is None:
            raise ValueError("UV unroll sensor requires a UVRemap instance.")
-        return UVUnrollFrameProcessor(uv_remap, **uv_kwargs)
+        return FrameUnwarper(uv_remap, **uv_kwargs)
    raise ValueError(f"Unsupported sensor type: {sensor_type}")


@@ -1,7 +1,7 @@
 """UV-based finger mesh unroll / flatten pipeline.

 Offline: Blender export -> build_or_load_domain -> unroll_map_uv.npz
-Runtime: live camera frame -> UVUnrollFrameProcessor -> Sensor (with valid_mask)
+Runtime: live camera frame -> FrameUnwarper -> Sensor (with valid_mask)
 3D display: UVDomain.surface.world_xyz
 """

@@ -4,7 +4,7 @@ import cv2
 import numpy as np
 import pytest

-from orisys.frame_processors import UVUnrollFrameProcessor
+from orisys.frame_processors import FrameUnwarper
 from orisys.unroll import UVDomain


@@ -24,7 +24,7 @@ def test_uv_unroll_processor_output_shape(uv_domain):
    src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
    src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))

-    processor = UVUnrollFrameProcessor(
+    processor = FrameUnwarper(
        uv_remap,
        source_width=src_w,
        source_height=src_h,
@@ -45,7 +45,7 @@ def test_uv_unroll_processor_prepare_source(uv_domain):
    src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
    src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))

-    processor = UVUnrollFrameProcessor(
+    processor = FrameUnwarper(
        uv_remap,
        source_width=src_w,
        source_height=src_h,
@@ -61,7 +61,7 @@ def test_uv_unroll_processor_flatten_prepared_matches_process(uv_domain):
    src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
    src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))

-    processor = UVUnrollFrameProcessor(
+    processor = FrameUnwarper(
        uv_remap,
        source_width=src_w,
        source_height=src_h,
@@ -78,7 +78,7 @@ def test_uv_unroll_processor_gray_fast_path_matches_bgr_result(uv_domain):
    src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
    src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))

-    processor = UVUnrollFrameProcessor(
+    processor = FrameUnwarper(
        uv_remap,
        source_width=src_w,
        source_height=src_h,
@@ -15,7 +15,7 @@ import cv2
 import numpy as np
 import orisys
 from orisys.configs import load_config
-from orisys.frame_processors import UVUnrollFrameProcessor
+from orisys.frame_processors import FrameUnwarper
 from orisys.unroll import (
    barycentric_grid,
    build_or_load_domain,
@@ -575,7 +575,7 @@ def main():
        f"tol={sensor.masked_poisson_tol:.1e})"
    )

-    frame_processor = UVUnrollFrameProcessor(
+    frame_processor = FrameUnwarper(
        domain.remap,
        camera_matrix=source_K if dist_coeffs is not None else None,
        dist_coeffs=dist_coeffs,
@@ -1,44 +0,0 @@
-import numpy as np
-from PIL import Image, ImageDraw
-
-# Parameters
-R = 9.0          # mm
-H = 20.0         # mm
-fov_half = np.radians(75)  # 75 degrees half-angle
-pixels_per_mm = 100
-
-# Texture dimensions
-width = int(np.pi * 2 * R * pixels_per_mm)   # circumference
-height = int(H * pixels_per_mm)
-
-# Desired dot pattern in image space (normalized)
-delta_theta = np.radians(10)      # 10° azimuth spacing
-delta_r = 0.1                     # radial spacing in normalized units
-rho = 0.08                        # normalized dot radius (from reference)
-
-# Generate dot centers in image space
-r_vals = np.arange(R/H, np.tan(fov_half), delta_r)
-theta_vals = np.arange(0, 2*np.pi, delta_theta)
-
-# Create empty texture (white background)
-img = Image.new('L', (width, height), color=255)
-draw = ImageDraw.Draw(img)
-
-for r_img in r_vals:
-    z = R / r_img
-    v = z / H
-    # Ellipse axes in texture coordinates
-    a_u = (rho * z) / (2 * np.pi * R)
-    a_v = (rho * z**2) / (R * H)
-    # Convert to pixels
-    a_u_px = a_u * width
-    a_v_px = a_v * height
-    for theta in theta_vals:
-        u = theta / (2 * np.pi)
-        cx = u * width
-        cy = v * height
-        # Draw ellipse (bounding box)
-        bbox = [cx - a_u_px, cy - a_v_px, cx + a_u_px, cy + a_v_px]
-        draw.ellipse(bbox, fill=0)   # black dot
-
-img.save('cylinder_marker_texture.png')
@@ -0,0 +1,233 @@
+#%%
+import numpy as np
+import cv2
+
+def merge_intervals(intervals):
+    # Merge overlapping intervals.
+    if not intervals:
+        return []
+    intervals.sort(key=lambda x: x[0])
+    merged = [intervals[0]]
+    for current in intervals[1:]:
+        last = merged[-1]
+        if current[0] <= last[1]:
+            merged[-1] = (last[0], max(last[1], current[1]))
+        else:
+            merged.append(current)
+    return merged
+
+def get_allowed_intervals(neighbor_values, delta):
+    # Compute forbidden intervals for this channel from neighbor values.
+    forbidden = []
+    for c in neighbor_values:
+        low = max(0, c - delta)
+        high = min(1, c + delta)
+        forbidden.append((low, high))
+    # Merge overlapping forbidden intervals.
+    merged = merge_intervals(forbidden)
+    allowed = []
+    # Allowed region before the first forbidden interval.
+    if merged:
+        if merged[0][0] > 0:
+            allowed.append((0, merged[0][0]))
+        # Allowed regions between forbidden intervals.
+        for i in range(1, len(merged)):
+            allowed.append((merged[i-1][1], merged[i][0]))
+        # Allowed region after the last forbidden interval.
+        if merged[-1][1] < 1:
+            allowed.append((merged[-1][1], 1))
+    else:
+        allowed.append((0, 1))
+    return allowed
+
+def sample_from_allowed(allowed):
+    # Choose a random value from the allowed intervals with probability proportional to interval length.
+    lengths = [end - start for start, end in allowed]
+    total_length = sum(lengths)
+    if total_length <= 0:
+        return None
+    point = np.random.rand() * total_length
+    cumulative = 0
+    for (start, end), length in zip(allowed, lengths):
+        cumulative += length
+        if point <= cumulative:
+            return np.random.uniform(start, end)
+    return allowed[-1][0]
+
+def sample_channel(neighbor_values, delta):
+    # Compute allowed intervals and sample a value.
+    allowed = get_allowed_intervals(neighbor_values, delta)
+    if not allowed:
+        return None
+    return sample_from_allowed(allowed)
+
+def clip_allowed_to_min(allowed, min_value):
+    clipped = []
+    for start, end in allowed:
+        start = max(start, min_value)
+        if end > start:
+            clipped.append((start, end))
+    return clipped
+
+
+def generate_valid_gray(neighbor_grays, r, black_threshold):
+    delta = np.sqrt(r)
+    min_gray = min_gray_normalized(black_threshold)
+    allowed = clip_allowed_to_min(get_allowed_intervals(neighbor_grays, delta), min_gray)
+    if not allowed:
+        return np.random.uniform(min_gray, 1.0)
+    sampled = sample_from_allowed(allowed)
+    if sampled is None:
+        return np.random.uniform(min_gray, 1.0)
+    return enforce_min_gray(sampled, black_threshold)
+
+
+def rgb_to_gray(color):
+    # ITU-R BT.601 luma, color channels in [0, 1].
+    r, g, b = color
+    return 0.299 * r + 0.587 * g + 0.114 * b
+
+
+def min_gray_normalized(black_threshold):
+    # Strictly above black_threshold in uint8 space.
+    return (float(black_threshold) + 1.0) / 255.0
+
+
+def gray_to_rgb(gray):
+    gray = float(gray)
+    return np.array([gray, gray, gray], dtype=np.float32)
+
+
+def enforce_min_gray(gray, black_threshold):
+    return max(float(gray), min_gray_normalized(black_threshold))
+
+
+def generate_valid_color(neighbors, r):
+    """
+    Generate a random color that satisfies, for each neighbor,
+    |R - R_neighbor| >= sqrt(r), and similarly for G and B.
+    neighbors: list of neighbor colors (each is an array of shape (3,))
+    r: threshold for squared difference.
+    """
+    delta = np.sqrt(r)
+    color = np.zeros(3)
+    for channel in range(3):
+        neighbor_channel = [n[channel] for n in neighbors]
+        sampled = sample_channel(neighbor_channel, delta)
+        # If the allowed intervals are empty, use a fallback.
+        if sampled is None:
+            sampled = np.random.rand()
+        color[channel] = sampled
+    return color
+
+def fill_pattern_remainder(pattern, h, w, d, black_threshold):
+    """Fill pixels not covered by full dxd patches (image size may not be divisible by d)."""
+    min_gray = min_gray_normalized(black_threshold)
+    filled_h = (h // d) * d
+    filled_w = (w // d) * d
+
+    if filled_h < h and filled_w > 0:
+        remainder_h = h - filled_h
+        source = pattern[filled_h - d:filled_h, :filled_w]
+        pattern[filled_h:h, :filled_w] = np.tile(
+            source, ((remainder_h + d - 1) // d, 1)
+        )[:remainder_h, :]
+
+    if filled_w < w and filled_h > 0:
+        remainder_w = w - filled_w
+        source = pattern[:filled_h, filled_w - d:filled_w]
+        pattern[:filled_h, filled_w:w] = np.tile(source, (1, (remainder_w + d - 1) // d))[:, :remainder_w]
+
+    if filled_h < h and filled_w < w:
+        remainder_h = h - filled_h
+        remainder_w = w - filled_w
+        source = pattern[filled_h - d:filled_h, filled_w - d:filled_w]
+        pattern[filled_h:h, filled_w:w] = np.tile(
+            source,
+            ((remainder_h + d - 1) // d, (remainder_w + d - 1) // d),
+        )[:remainder_h, :remainder_w]
+
+    pattern[:] = np.maximum(pattern, min_gray)
+    return pattern
+
+
+def generate_color_pattern(h, w, d, r, black_threshold):
+    """
+    Generate a dense grayscale pattern.
+    h, w: height and width of the output image in pixels.
+    d: patch size in pixels (each patch is d x d).
+    r: randomness regulation factor. For each neighbor the condition is:
+       (|channel_new - channel_neighbor|)^2 >= r.
+    black_threshold: minimum grayscale value in [0, 255]; patches cannot be black.
+    """
+    pattern = np.zeros((h, w), dtype=np.float32)
+    num_patches_x = h // d
+    num_patches_y = w // d
+    min_gray = min_gray_normalized(black_threshold)
+
+    for i in range(num_patches_x):
+        for j in range(num_patches_y):
+            # Collect neighbor patch grays as RGB triples for the existing sampler.
+            neighbors = []
+            neighbor_grays = []
+            if i > 0:
+                patch_above = pattern[(i-1)*d:i*d, j*d:(j+1)*d]
+                gray_above = float(np.mean(patch_above))
+                neighbor_grays.append(gray_above)
+                neighbors.append(gray_to_rgb(gray_above))
+            if j > 0:
+                patch_left = pattern[i*d:(i+1)*d, (j-1)*d:j*d]
+                gray_left = float(np.mean(patch_left))
+                neighbor_grays.append(gray_left)
+                neighbors.append(gray_to_rgb(gray_left))
+
+            if len(neighbors) == 0:
+                # First patch: sample directly above the black threshold.
+                gray = np.random.uniform(min_gray, 1.0)
+            else:
+                new_color = generate_valid_color(neighbors, r)
+                gray = rgb_to_gray(new_color)
+                if gray < min_gray:
+                    gray = generate_valid_gray(neighbor_grays, r, black_threshold)
+                else:
+                    gray = enforce_min_gray(gray, black_threshold)
+
+            pattern[i*d:(i+1)*d, j*d:(j+1)*d] = gray
+            print('Patch ({}, {}) gray={:.3f}'.format(i, j, gray))
+
+    return fill_pattern_remainder(pattern, h, w, d, black_threshold)
+
+def main():
+    # Parameters for the pattern.
+    # For example, using 100x100 pixels image with patches of 3x3 pixels.
+    # height = 2750
+    # width = 1950
+    height = 1340
+    width = 1360
+    patch_size = 12
+    black_threshold = 125
+    # Set r to a value that gives sufficient difference.
+    # Note: r is the threshold for squared difference per channel, and the condition is
+    # |c - c_neighbor| >= sqrt(r). Very high r may make it impossible to satisfy.
+    r = 1000
+
+    pattern = generate_color_pattern(height, width, patch_size, r, black_threshold)
+    # Scale to 0-255 for display and saving.
+    pattern_display = (pattern * 255).astype(np.uint8)
+
+    if pattern_display.min() <= black_threshold:
+        raise RuntimeError(
+            f"Generated pattern contains values <= black_threshold ({black_threshold}); "
+            f"min={int(pattern_display.min())}"
+        )
+
+    # Display the pattern with OpenCV.
+    cv2.imshow('Pattern', pattern_display)
+    cv2.waitKey(0)
+    cv2.destroyAllWindows()
+
+    # Save the pattern as an image file.
+    cv2.imwrite('gray_pattern.png', pattern_display)
+
+if __name__ == '__main__':
+    main()