Dev #19

Merged
zhangyijian merged 53 commits from dev into main 2026-09-02 21:39:04 +08:00
14 changed files with 275 additions and 94 deletions
Showing only changes of commit 30f9deb5b2 - Show all commits
+2 -2
View File
@@ -2,10 +2,10 @@
a = Analysis(
['D:\\repo\\orisyssdk_git\\examples\\finger_pipeline_3dview.py'],
['D:\\repo\\orisys-sdk\\examples\\finger_pipeline_3dview.py'],
pathex=[],
binaries=[],
datas=[('D:\\repo\\orisyssdk_git\\config', 'config'), ('D:\\repo\\orisyssdk_git\\assets', 'assets')],
datas=[('D:\\repo\\orisys-sdk\\config', 'config'), ('D:\\repo\\orisys-sdk\\assets', 'assets')],
hiddenimports=['PyQt5', 'pyqtgraph', 'open3d', 'cv2', 'orisys', 'gui', 'pipeline', 'viz', 'app_paths', 'plotly', '_plotly_utils', 'open3d.cpu.pybind'],
hookspath=[],
hooksconfig={},
+7 -16
View File
@@ -54,15 +54,6 @@ import orisys
print(orisys.__version__)
```
5 安装opencl(实验):
在不带nvidia显卡的设备,可选择通过opencl进行gpu 计算加速,需要确保opencl的库或驱动已经安装,然后在conda环境中安装依赖:
``` bash
pip install pyopencl
conda install conda-forge::pyvkfft
```
***
## `Sensor` 类别
@@ -245,32 +236,32 @@ flow = sensor.read_info(sensor.info.VRAW)
***
***
## `orisys.finger` 指纹 UV pipeline
## `orisys.finger` 手指 UV pipeline
`examples/finger_pipeline.py` 展示了一个更适合指纹/单相机场景的高层 API:
`examples/finger_pipeline.py` 展示了一个更适合手指/单相机场景的高层 API:
1. 使用 `FingerRuntime.open()` 打开视频源、加载 Blender 导出的资产,并构建 UV 展开映射。
2. 用 `grab_and_process()` 或 `read_source() + process_frame()` 获取一帧展开后的触觉图像。
3. 通过 `runtime.tactile`(`TactileProcessor`)调用 `compute_deformation()`、`compute_contact()`、`read_info()`。
4. 退出前调用 `runtime.close()` 释放视频源和内部 `Sensor` 资源。
相比直接使用 `Sensor`,`FingerRuntime` 已封装了视频读取、相机尺寸设置、UV 展开和有效区域掩码传递,更适合作为指纹 2D/3D 示例与上层应用的入口。
相比直接使用 `Sensor`,`FingerRuntime` 已封装了视频读取、相机尺寸设置、UV 展开和有效区域掩码传递,更适合作为手指 2D/3D 示例与上层应用的入口。
### 1. `FingerRuntime.open` 方法
#### 描述
创建一个可复用的指纹 UV runtime。该方法会完成以下初始化:
创建一个可复用的手指 UV runtime。该方法会完成以下初始化:
- 解析 `export_dir` 下的 `mesh.npz`、`camera_intrinsics.npz`、`world_to_cam.npy`
- 根据配置和标定信息构建/加载 `unroll_map_uv.npz`
- 打开相机或视频文件
- 创建内部 `Sensor("external", ...)`
- 创建 `UVUnrollFrameProcessor` 与 `TactileProcessor`
- 创建 `FrameUnwarper` 与 `TactileProcessor`
#### 输入参数
* **export_dir** (`str`,可选): 指纹导出资产目录,默认值来自 `orisys.finger.DEFAULT_FINGER_EXPORT_DIR`。目录下通常需要包含 `mesh.npz`、`camera_intrinsics.npz`、`world_to_cam.npy`。
* **export_dir** (`str`,可选): 手指导出资产目录,默认值来自 `orisys.finger.DEFAULT_FINGER_EXPORT_DIR`。目录下通常需要包含 `mesh.npz`、`camera_intrinsics.npz`、`world_to_cam.npy`。
* **config** (`str`,可选): Orisys 配置文件路径,默认 `./config/dd02-ov.json`。
* **map_cache** (`str`,可选): UV 展开缓存文件路径,默认值来自 `orisys.finger.DEFAULT_FINGER_MAP_CACHE`。
* **source_scale** (`float`,可选): 输入相机分辨率缩放比例;未提供时根据配置自动推断。
@@ -402,7 +393,7 @@ runtime = FingerRuntime.open(
除 `push_frame()` / `warmup()` 之外,`TactileProcessor` 的 `compute_deformation()`、`compute_contact()`、`read_info()`、`reset()` 调用方式与 `Sensor` 基本一致,因此可直接复用前文 `Sensor` 章节中的参数说明。
### 10. 最小指纹 pipeline 示例
### 10. 最小手指 pipeline 示例
下面的流程与 `examples/finger_pipeline.py` 一致:
+17 -17
View File
@@ -1,16 +1,16 @@
叠动科技软件使用手册
版本: 0.4.1
更新日期: 2026年4月16日
更新日期: 2026年7月13日
# 简介
本文件介绍如何上手和运行叠动视触觉传感器配套的开发套件。本套件是配合叠动视触觉传感器进行图像处理,计算形变、力以及接触信息。它可以从摄像头或视频文件中读取图像,处理后打印和展示处理后信息。
支持操作系统:Windows、Ubuntu
建议Python 版本: 3.10
建议Python 版本: 3.10, 3.11, 3.12
# 基础环境设置
## 安装 Python 与 Anaconda
1. 访问 [Anaconda 官网](https://www.anaconda.com/download)下载适合您操作系统的安装包(Windows、Linux)。
2. 按照提示完成安装。(Widnows:建议在安装过程中勾选“将 Anaconda 添加到系统环境变量”选项,方便后续使用)。
2. 按照提示完成安装。
3. 安装完成后,打开命令行终端(Windows 用户可在”Anaconda Navigator”打开“Anaconda Prompt”),输入 `conda info` ,若能显示 conda 的信息,则说明安装成功。
## 创建conda虚拟环境
@@ -115,8 +115,8 @@ print(orisys.__version__)
套件里面包含了多个例子,建议都在仓库根目录运行(即 `config/`、`examples/`、`assets/` 同级目录)。常见入口分为两类:
1. `examples/basic_pipeline.py`:复眼/拼接传感器基础 OpenCV 示例
2. `examples/finger_pipeline.py`:指纹传感器 UV 展开最小流程示例
3. `examples/finger_pipeline_3dview.py`:指纹传感器 Qt 3D 可视化界面
2. `examples/finger_pipeline.py`:手指传感器2维显示最小流程示例
3. `examples/finger_pipeline_3dview.py`:手指传感器 Qt 3D 可视化界面
下面先介绍最基础的 `basic_pipeline.py` 运行方式。
@@ -140,9 +140,9 @@ print(orisys.__version__)
```
5. 程序会运行,并自动打开可视化窗口和打印资料,在窗口上按键盘”q”键可以停止运行,按”r”键可以重设数据
### 指纹传感器 UV 示例
### 手指传感器 UV 示例
如果您使用的是指纹/单相机 UV 展开流程,可运行:
如果您使用的是手指/单相机 UV 展开流程,可运行:
``` bash
python examples/finger_pipeline.py --camera_id 0 --config ./config/dd02-ov.json
@@ -164,30 +164,30 @@ python examples/finger_pipeline.py --video ./data/sample.mp4 --config ./config/d
常用参数:
- `--verbose`:输出更详细的初始化日志
- `--profile`:打印每个阶段的耗时统计
- `--profile-every N`:每 N 帧输出一次平均耗时
例如:
``` bash
python examples/finger_pipeline.py --camera_id 0 --config ./config/dd02-ov.json --profile --profile-every 30
python examples/finger_pipeline.py --camera_id 0 --config ./config/dd02-ov.json
```
### 指纹 3D 可视化界面
如果已经安装 GUI 依赖,还可以运行:
### 手指 3D 可视化界面
如果要使用 GUI ,安装GUI依赖:
``` bash
pip install open3d
pip install pyqt5 pyqtgraph
```
然后运行:
``` bash
pip install -e ".[gui]"
python examples/finger_pipeline_3dview.py --video 0 --config ./config/dd02-ov.json
```
该界面会提供:
- 指纹 UV 展开图
- 手指平面化光流箭头显示
- 力曲线
- 3D mesh 视图
- 录制与重置等交互控件
- 3D 视图
## 开发套件使用方法
参考开发套件附带的开发文档。
+1 -1
View File
@@ -3,7 +3,7 @@
Minimum UV-unroll finger tactile pipeline.
Offline: load Blender export -> build_or_load_domain -> unroll_map_uv.npz
Runtime: camera frame -> UVUnrollFrameProcessor -> TactileProcessor -> forces/contact
Runtime: camera frame -> FrameUnwarper -> TactileProcessor -> forces/contact
For the full demo (UV debug panels, dashboard, benchmarks) see:
tools/3d_patch_generation_blender/unroll_realtime.py
+2 -1
View File
@@ -9,4 +9,5 @@ scipy==1.13
setuptools>=45.2.0
av
numba
pyqt5
pyqtgraph
+3 -3
View File
@@ -6,7 +6,7 @@ import cv2
import numpy as np
from ..configs import load_config
from ..frame_processors import UVUnrollFrameProcessor
from ..frame_processors import FrameUnwarper
from ..sensor import Sensor
from ..unroll import (
build_or_load_domain,
@@ -208,7 +208,7 @@ class FingerRuntime:
geometry: FingerGeometry,
domain,
tactile: TactileProcessor,
frame_processor: UVUnrollFrameProcessor,
frame_processor: FrameUnwarper,
capture,
):
self.geometry = geometry
@@ -268,7 +268,7 @@ class FingerRuntime:
raise RuntimeError("Cannot open camera/video source")
sensor = Sensor("external", config_name=config, verbose=verbose, mirror=mirror)
frame_processor = UVUnrollFrameProcessor(
frame_processor = FrameUnwarper(
domain.remap,
camera_matrix=geometry.source_K if geometry.dist_coeffs is not None else None,
dist_coeffs=geometry.dist_coeffs,
+2 -2
View File
@@ -27,7 +27,7 @@ class SingleCameraFrameProcessor:
return frame
class UVUnrollFrameProcessor:
class FrameUnwarper:
"""Remap a raw camera frame into a flattened UV-domain BGR image."""
def __init__(
@@ -227,5 +227,5 @@ def create_frame_processor(sensor_type, stitcher=None, uv_remap=None, **uv_kwarg
if sensor_type in ("uv_unroll", "unroll"):
if uv_remap is None:
raise ValueError("UV unroll sensor requires a UVRemap instance.")
return UVUnrollFrameProcessor(uv_remap, **uv_kwargs)
return FrameUnwarper(uv_remap, **uv_kwargs)
raise ValueError(f"Unsupported sensor type: {sensor_type}")
+1 -1
View File
@@ -1,7 +1,7 @@
"""UV-based finger mesh unroll / flatten pipeline.
Offline: Blender export -> build_or_load_domain -> unroll_map_uv.npz
Runtime: live camera frame -> UVUnrollFrameProcessor -> Sensor (with valid_mask)
Runtime: live camera frame -> FrameUnwarper -> Sensor (with valid_mask)
3D display: UVDomain.surface.world_xyz
"""
+5 -5
View File
@@ -4,7 +4,7 @@ import cv2
import numpy as np
import pytest
from orisys.frame_processors import UVUnrollFrameProcessor
from orisys.frame_processors import FrameUnwarper
from orisys.unroll import UVDomain
@@ -24,7 +24,7 @@ def test_uv_unroll_processor_output_shape(uv_domain):
src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))
processor = UVUnrollFrameProcessor(
processor = FrameUnwarper(
uv_remap,
source_width=src_w,
source_height=src_h,
@@ -45,7 +45,7 @@ def test_uv_unroll_processor_prepare_source(uv_domain):
src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))
processor = UVUnrollFrameProcessor(
processor = FrameUnwarper(
uv_remap,
source_width=src_w,
source_height=src_h,
@@ -61,7 +61,7 @@ def test_uv_unroll_processor_flatten_prepared_matches_process(uv_domain):
src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))
processor = UVUnrollFrameProcessor(
processor = FrameUnwarper(
uv_remap,
source_width=src_w,
source_height=src_h,
@@ -78,7 +78,7 @@ def test_uv_unroll_processor_gray_fast_path_matches_bgr_result(uv_domain):
src_h = int(uv_remap.metadata.get("source_height", uv_remap.out_height))
src_w = int(uv_remap.metadata.get("source_width", uv_remap.out_width))
processor = UVUnrollFrameProcessor(
processor = FrameUnwarper(
uv_remap,
source_width=src_w,
source_height=src_h,
Binary file not shown.
Binary file not shown.
@@ -15,7 +15,7 @@ import cv2
import numpy as np
import orisys
from orisys.configs import load_config
from orisys.frame_processors import UVUnrollFrameProcessor
from orisys.frame_processors import FrameUnwarper
from orisys.unroll import (
barycentric_grid,
build_or_load_domain,
@@ -575,7 +575,7 @@ def main():
f"tol={sensor.masked_poisson_tol:.1e})"
)
frame_processor = UVUnrollFrameProcessor(
frame_processor = FrameUnwarper(
domain.remap,
camera_matrix=source_K if dist_coeffs is not None else None,
dist_coeffs=dist_coeffs,
@@ -1,44 +0,0 @@
import numpy as np
from PIL import Image, ImageDraw
# Parameters
R = 9.0 # mm
H = 20.0 # mm
fov_half = np.radians(75) # 75 degrees half-angle
pixels_per_mm = 100
# Texture dimensions
width = int(np.pi * 2 * R * pixels_per_mm) # circumference
height = int(H * pixels_per_mm)
# Desired dot pattern in image space (normalized)
delta_theta = np.radians(10) # 10° azimuth spacing
delta_r = 0.1 # radial spacing in normalized units
rho = 0.08 # normalized dot radius (from reference)
# Generate dot centers in image space
r_vals = np.arange(R/H, np.tan(fov_half), delta_r)
theta_vals = np.arange(0, 2*np.pi, delta_theta)
# Create empty texture (white background)
img = Image.new('L', (width, height), color=255)
draw = ImageDraw.Draw(img)
for r_img in r_vals:
z = R / r_img
v = z / H
# Ellipse axes in texture coordinates
a_u = (rho * z) / (2 * np.pi * R)
a_v = (rho * z**2) / (R * H)
# Convert to pixels
a_u_px = a_u * width
a_v_px = a_v * height
for theta in theta_vals:
u = theta / (2 * np.pi)
cx = u * width
cy = v * height
# Draw ellipse (bounding box)
bbox = [cx - a_u_px, cy - a_v_px, cx + a_u_px, cy + a_v_px]
draw.ellipse(bbox, fill=0) # black dot
img.save('cylinder_marker_texture.png')
@@ -0,0 +1,233 @@
#%%
import numpy as np
import cv2
def merge_intervals(intervals):
# Merge overlapping intervals.
if not intervals:
return []
intervals.sort(key=lambda x: x[0])
merged = [intervals[0]]
for current in intervals[1:]:
last = merged[-1]
if current[0] <= last[1]:
merged[-1] = (last[0], max(last[1], current[1]))
else:
merged.append(current)
return merged
def get_allowed_intervals(neighbor_values, delta):
# Compute forbidden intervals for this channel from neighbor values.
forbidden = []
for c in neighbor_values:
low = max(0, c - delta)
high = min(1, c + delta)
forbidden.append((low, high))
# Merge overlapping forbidden intervals.
merged = merge_intervals(forbidden)
allowed = []
# Allowed region before the first forbidden interval.
if merged:
if merged[0][0] > 0:
allowed.append((0, merged[0][0]))
# Allowed regions between forbidden intervals.
for i in range(1, len(merged)):
allowed.append((merged[i-1][1], merged[i][0]))
# Allowed region after the last forbidden interval.
if merged[-1][1] < 1:
allowed.append((merged[-1][1], 1))
else:
allowed.append((0, 1))
return allowed
def sample_from_allowed(allowed):
# Choose a random value from the allowed intervals with probability proportional to interval length.
lengths = [end - start for start, end in allowed]
total_length = sum(lengths)
if total_length <= 0:
return None
point = np.random.rand() * total_length
cumulative = 0
for (start, end), length in zip(allowed, lengths):
cumulative += length
if point <= cumulative:
return np.random.uniform(start, end)
return allowed[-1][0]
def sample_channel(neighbor_values, delta):
# Compute allowed intervals and sample a value.
allowed = get_allowed_intervals(neighbor_values, delta)
if not allowed:
return None
return sample_from_allowed(allowed)
def clip_allowed_to_min(allowed, min_value):
clipped = []
for start, end in allowed:
start = max(start, min_value)
if end > start:
clipped.append((start, end))
return clipped
def generate_valid_gray(neighbor_grays, r, black_threshold):
delta = np.sqrt(r)
min_gray = min_gray_normalized(black_threshold)
allowed = clip_allowed_to_min(get_allowed_intervals(neighbor_grays, delta), min_gray)
if not allowed:
return np.random.uniform(min_gray, 1.0)
sampled = sample_from_allowed(allowed)
if sampled is None:
return np.random.uniform(min_gray, 1.0)
return enforce_min_gray(sampled, black_threshold)
def rgb_to_gray(color):
# ITU-R BT.601 luma, color channels in [0, 1].
r, g, b = color
return 0.299 * r + 0.587 * g + 0.114 * b
def min_gray_normalized(black_threshold):
# Strictly above black_threshold in uint8 space.
return (float(black_threshold) + 1.0) / 255.0
def gray_to_rgb(gray):
gray = float(gray)
return np.array([gray, gray, gray], dtype=np.float32)
def enforce_min_gray(gray, black_threshold):
return max(float(gray), min_gray_normalized(black_threshold))
def generate_valid_color(neighbors, r):
"""
Generate a random color that satisfies, for each neighbor,
|R - R_neighbor| >= sqrt(r), and similarly for G and B.
neighbors: list of neighbor colors (each is an array of shape (3,))
r: threshold for squared difference.
"""
delta = np.sqrt(r)
color = np.zeros(3)
for channel in range(3):
neighbor_channel = [n[channel] for n in neighbors]
sampled = sample_channel(neighbor_channel, delta)
# If the allowed intervals are empty, use a fallback.
if sampled is None:
sampled = np.random.rand()
color[channel] = sampled
return color
def fill_pattern_remainder(pattern, h, w, d, black_threshold):
"""Fill pixels not covered by full dxd patches (image size may not be divisible by d)."""
min_gray = min_gray_normalized(black_threshold)
filled_h = (h // d) * d
filled_w = (w // d) * d
if filled_h < h and filled_w > 0:
remainder_h = h - filled_h
source = pattern[filled_h - d:filled_h, :filled_w]
pattern[filled_h:h, :filled_w] = np.tile(
source, ((remainder_h + d - 1) // d, 1)
)[:remainder_h, :]
if filled_w < w and filled_h > 0:
remainder_w = w - filled_w
source = pattern[:filled_h, filled_w - d:filled_w]
pattern[:filled_h, filled_w:w] = np.tile(source, (1, (remainder_w + d - 1) // d))[:, :remainder_w]
if filled_h < h and filled_w < w:
remainder_h = h - filled_h
remainder_w = w - filled_w
source = pattern[filled_h - d:filled_h, filled_w - d:filled_w]
pattern[filled_h:h, filled_w:w] = np.tile(
source,
((remainder_h + d - 1) // d, (remainder_w + d - 1) // d),
)[:remainder_h, :remainder_w]
pattern[:] = np.maximum(pattern, min_gray)
return pattern
def generate_color_pattern(h, w, d, r, black_threshold):
"""
Generate a dense grayscale pattern.
h, w: height and width of the output image in pixels.
d: patch size in pixels (each patch is d x d).
r: randomness regulation factor. For each neighbor the condition is:
(|channel_new - channel_neighbor|)^2 >= r.
black_threshold: minimum grayscale value in [0, 255]; patches cannot be black.
"""
pattern = np.zeros((h, w), dtype=np.float32)
num_patches_x = h // d
num_patches_y = w // d
min_gray = min_gray_normalized(black_threshold)
for i in range(num_patches_x):
for j in range(num_patches_y):
# Collect neighbor patch grays as RGB triples for the existing sampler.
neighbors = []
neighbor_grays = []
if i > 0:
patch_above = pattern[(i-1)*d:i*d, j*d:(j+1)*d]
gray_above = float(np.mean(patch_above))
neighbor_grays.append(gray_above)
neighbors.append(gray_to_rgb(gray_above))
if j > 0:
patch_left = pattern[i*d:(i+1)*d, (j-1)*d:j*d]
gray_left = float(np.mean(patch_left))
neighbor_grays.append(gray_left)
neighbors.append(gray_to_rgb(gray_left))
if len(neighbors) == 0:
# First patch: sample directly above the black threshold.
gray = np.random.uniform(min_gray, 1.0)
else:
new_color = generate_valid_color(neighbors, r)
gray = rgb_to_gray(new_color)
if gray < min_gray:
gray = generate_valid_gray(neighbor_grays, r, black_threshold)
else:
gray = enforce_min_gray(gray, black_threshold)
pattern[i*d:(i+1)*d, j*d:(j+1)*d] = gray
print('Patch ({}, {}) gray={:.3f}'.format(i, j, gray))
return fill_pattern_remainder(pattern, h, w, d, black_threshold)
def main():
# Parameters for the pattern.
# For example, using 100x100 pixels image with patches of 3x3 pixels.
# height = 2750
# width = 1950
height = 1340
width = 1360
patch_size = 12
black_threshold = 125
# Set r to a value that gives sufficient difference.
# Note: r is the threshold for squared difference per channel, and the condition is
# |c - c_neighbor| >= sqrt(r). Very high r may make it impossible to satisfy.
r = 1000
pattern = generate_color_pattern(height, width, patch_size, r, black_threshold)
# Scale to 0-255 for display and saving.
pattern_display = (pattern * 255).astype(np.uint8)
if pattern_display.min() <= black_threshold:
raise RuntimeError(
f"Generated pattern contains values <= black_threshold ({black_threshold}); "
f"min={int(pattern_display.min())}"
)
# Display the pattern with OpenCV.
cv2.imshow('Pattern', pattern_display)
cv2.waitKey(0)
cv2.destroyAllWindows()
# Save the pattern as an image file.
cv2.imwrite('gray_pattern.png', pattern_display)
if __name__ == '__main__':
main()