Compare commits

..

53 Commits

Author SHA1 Message Date
ykcheungab 9c951e9d3c fix: 将 ARM64 wheel 纳入发布校验和上传流程
补齐全部架构的运行证据、GitHub 发布说明和 PyPI 逐文件下载核验。已通过 161 项测试、代码和文档门禁、九个 wheel 的本地索引下载核验,以及 ARM 三个 ABI 的 40 帧 CPU 检查。
2026-10-15 21:39:31 +08:00
ykcheungab bb15b750fb feat: 添加 ARM64 manylinux wheel 构建选项
复用现有 Docker 和 Nuitka 流程,隔离架构镜像并支持 aarch64 标签。已通过 47 项测试、代码及文档门禁,并完成 Python 3.10 ARM64 wheel 构建、隔离导入与内容审计。
2026-10-14 16:08:54 +08:00
ykcheungab 3953747f17 updated sensor config for better visualization 2026-10-14 13:52:24 +08:00
ykcheungab ae64c65c45 调整 SDK 发布同步清单并修正文档版本约束
移除已删除的标定目录同步项,避免同步失败;将 Ruff 版本写成依赖约束,避免被误判为 SDK 版本。快速回归398项通过,六个 wheel 通过静态及视频检查。
2026-10-11 21:19:40 +08:00
ykcheungab 120b8c2cf6 为采集帧绑定主机时间与来源信息
采集时间随图像穿过处理队列,并修正未读帧的丢帧计数。通过采集与流水线针对性测试。
2026-10-11 12:48:34 +08:00
ykcheungab 0784eacf51 把 FN/FX/FY 力读出抽到 ForceReadout。
滤波预设、EMA、中值、低通与标定换算离开 Sensor;对外 configure / filters_state 不变。已有 signal_processing、filter_presets、calibration 测试通过。
2026-10-08 10:08:35 +08:00
ykcheungab 68edb0dc4c 去掉与 examples/README.md 重复的示例走查
本文件 L184-265 重复了 examples/README.md 已有的内容:示例清单、各产品入口、全部命令与参数、3dview 说明。删掉重复段落改为指向该文件,只保留首次运行所需的最短路径,并把 Windows/Ubuntu 的取 ID 步骤与运行结果合并,不再各写一遍。

验证:doc_count 1439/1775,doc_lint changed 与 --all 均干净。
2026-10-06 14:27:56 +08:00
ykcheungab 4c3d0f8864 updated get_started.md to match new SDK calibration data loading and usage. 2026-10-05 14:07:05 +08:00
ykcheungab ceb06f5702 为基础示例接入按设备 ID 选择的力标定
让使用者从本地目录选标定并看见实际单位。短片回放及相关测试通过。
2026-10-05 14:02:40 +08:00
ykcheungab 8c147f03aa 新增 SDK 贡献者英文入门与发布指南
让源码开发和发布交接有清晰入口,并保留现有中文文档。文档门禁与标定测试通过。
2026-10-05 13:29:36 +08:00
ykcheungab 34cb83d4a4 修正示例中的三轴力单位显示
避免未标定读数被示例标成 N,并让 Finger 示例读取 SDK 的每轴单位。已通过相关测试和语法检查。
2026-10-05 13:10:58 +08:00
ykcheungab 38d3e67b00 固定 HHD 下采样并标记三轴力单位
避免运行中改变 HHD 测量配置,并明确部分标定时各轴单位。已通过 SDK 与共享层测试。
2026-10-05 12:56:29 +08:00
ykcheungab 80f7da5a20 添加按设备 ID 选择和校验标定文件
避免多台传感器误用默认标定,缺文件时保持 raw。已通过 SDK 回归测试和文档检查。
2026-10-05 12:21:04 +08:00
ykcheungab 230b04a79d Add a sensor bundle contract separating family, backend and profile
Product name decided the pipeline: Mini worked live but replay refused it. A bundle descriptor now declares profile_id, product_family and processing_backend, with computational and visualization assets kept apart, and one resolver serves live and recorded input. Legacy bundles load through a documented adapter that never guesses a backend from a folder name.
2026-10-04 20:55:39 +08:00
ykcheungab 6e3d509239 Add pipeline pause and resume at two levels
Pause now holds the pipeline without tearing it down: CAPTURE_ONLY keeps the camera
streaming so a live sensor stays converged while the flow and HHD stages stop, and BOTH stops
the read as well. Resume rebases the tracker, because a held reference frame would otherwise
report the hold itself as deformation.

Verified with tests/test_pipeline_pause.py: 10 passed across three consecutive runs.
2026-10-04 20:55:35 +08:00
ykcheungab 333fa8fcff added pause/wait functions for stream 2026-09-30 21:23:08 +08:00
ykcheungab f700102067 docs: record the GUI extraction plan, postponed
examples/gui/ is a full Qt + open3d application (30 modules, 4,815 lines) rather
than an example, and its proper home is the orisys-studio application. The move
is planned but not started, so this records the plan rather than performing any
of it.

What the plan establishes, so the work does not have to be re-derived:

- src/orisys/ imports no gui/viz/examples module, verified by AST rather than
  grep. The dependency is one-way, so the GUI can leave without a library
  change. The single apparent hit is `from .pipeline import ...` in
  src/orisys/__init__.py, which is the SDK's own module.
- examples/viz/ is shared and cannot follow the GUI wholesale: it mixes open3d
  renderers with toolkit-independent helpers that examples/pipeline/service.py
  and tests/test_contact_arrow_scale.py depend on. It has to be split.
- The real coupling is inverted and lives in the tests: tests/test_qt_viewer_backend.py
  imports examples/gui, and tests/conftest.py deliberately puts examples/ ahead
  of build/ because both define a package named `gui`. Removing the GUI means
  re-deriving that guard, not deleting it blindly.
- UVMeshViewer and friends are in src/orisys/finger/__init__.__all__, so removing
  them is a breaking change to a released API and needs a deprecation release
  first. It must not be folded into a code move.
- The `examples/` directory must keep existing in the packaged layout even after
  gui/ leaves it: src/orisys/configs.py anchors its run-root detection on a
  sibling examples/ folder.

Also updates examples/README.md to say "lives here for now" instead of "is moving
out", so nothing reads as work in progress. The GUI itself is untouched, and the
only guidance offered is that new SDK work should not build on it.
2026-09-30 17:10:17 +08:00
ykcheungab b5d7533e24 examples: make StreamPipeline the recommended entry, archive the manual loop
The two canonical examples taught the advanced path: they sequenced
get_img() -> compute_deformation() -> compute_contact() -> read_info() by hand
and unpacked a 13-element tuple positionally, which is what led developers to
reach for the compute backend, the capture queue and the stitch mode. The
managed alternative already existed and was documented nowhere.

Library (add-only, nothing removed):

- FrameSnapshot.image_size_raw carries the capture resolution, so a caller can
  compare it with the processed frame instead of reaching into Sensor.
- StreamPipeline.reset() resets the tracker and force filters *safely*. It is
  queued and applied by the flow stage before its next frame, never inline:
  Sensor.reset rebases initial_img on the current Sensor.img, which the flow
  stage is reading concurrently, so a caller-thread reset would capture a
  half-written reference frame. Idempotent.
- TactileProcessor.pipeline(produce) and FingerRuntime.pipeline() build a
  pipeline without the caller naming a backend concept; the latter is
  `with runtime.pipeline(contact=True) as pipe`.

Examples:

- examples/basic_pipeline.py and examples/finger_pipeline.py rewritten onto
  StreamPipeline + FrameSnapshot: one object, one call per frame, no tuple, and
  no compute backend, queue or stitch setting in sight.
- examples/archive/basic_pipeline_manual.py and _finger_pipeline_manual.py keep
  the previous versions byte-for-byte apart from a supersede header, so code
  copied from them keeps working. The finger header also records the archived
  --show defect (store_true with default=True, so it could never be disabled).
- examples/_display.py holds the shared display helpers, so an example no longer
  hand-rolls depth normalisation and colormap application.
- examples/README.md leads with the recommended pair and documents the manual
  form as still-supported, and states that examples/gui/ is moving to
  orisys-studio.

Tests: three added to tests/test_pipeline.py, covering the reset queue (it must
not run on the caller's thread and must land before the first compute_flow),
image_size_raw, and the None case when a producer reports no size. Verified the
reset test fails when reset() is made inline, so it discriminates.

Verified end-to-end against real media: basic_pipeline on data/sample/s1_480p.mp4
(box) reaches ~113 FPS and exits 0; finger_pipeline on data/dd02-0703/1.mp4
(finger) reaches ~140 FPS with mask coverage 0.41 and exits 0. Both archived
manual versions still run. tests/test_pipeline.py: 20 passed. Full suite:
342 passed, 3 failed - all three failures reproduce on the pristine tree
(import precedence, longrun timing, and a stale ruff version literal in
docs/code-review.md) and are unrelated to this change.
2026-09-30 17:09:02 +08:00
ykcheungab 4c1ab5c27e tools: add a static public-API audit and record its first run
Adds tools/api_audit.py, a read-only static audit of the public API surface,
and docs/api-audit.md, its first run against this revision.

The audit answers three questions the reference cannot answer about itself:

1. Does every declared export exist, have a docstring, and appear in
   docs/api.md? It reads the root __all__ plus each submodule's __all__.
2. How complete is each public definition? Docstring presence, detected
   docstring convention, parameter and return annotations.
3. Which public names do examples and tests actually use but the reference
   never explains? That set is the real documentation gap.

It parses with ast and never imports the package, so it costs no numpy/cv2
import and cannot be affected by the environment.

Findings on this revision (full detail in docs/api-audit.md):

- 393 public classes/methods/functions; 65% have a docstring, 32% have a fully
  annotated signature, 38 document their return value.
- 243 docstrings are unstructured prose, 137 are absent, 13 use Google or NumPy
  sections. There is effectively one convention: none.
- 9 of the 18 root exports are absent from docs/api.md, including
  StreamPipeline, StreamRecorder, mosaic_videos and FFCFrameProcessor - the
  pipelined multi-stream and offline-recording paths.
- Six sub-surfaces have zero coverage in the reference: unroll (18 exports),
  capture (17), hhd + solvers (7), pipeline (3), export (2).
- 74 names are used by examples or tests but absent from the reference.
- 21 public classes are exported nowhere; LowpassFilter is documented in
  docs/api.md and importable, but declared in no __all__.

No behaviour change: a new tool and a new document, no source file touched.
2026-09-30 16:44:12 +08:00
ykcheungab cde832eaca tools: delegate documentation checks to the workspace doc gate
`tools/lint_gate.py` now also runs `D:\orisys_ws\scripts\doc_lint.py`, the
workspace's single documentation gate, alongside Ruff:

- changed Markdown by default, the whole corpus under --all
- `--files` names Python files explicitly, so it skips the doc gate
- `--no-docs` opts out
- the docs verdict is reported even when Ruff is missing, so a missing
  linter cannot hide it (the exit code still signals the environment problem)

The doc rules live in exactly one place on purpose: this file is duplicated
across three repositories and had already drifted, so adding a fourth copy of
lint logic would repeat the mistake. Failing checks are the documentation
budget, inventory membership, a declared kind, unresolved mojibake, paragraph
walls, and duplicate explanations across documents. Emphasis density is
advisory only - measured as a continuum from 12% to 48% across the corpus, so
no threshold separates signal from noise.

Pre-existing oversized documents are recorded as budget-exempt with a written
reason in D:\orisys_ws\docs\inventory.json, and pre-existing findings are
baselined there, so this gate is red only for new regression.

With Ruff unavailable in the current environment the doc half still reports,
which is how it was verified.
2026-09-30 16:12:30 +08:00
ykcheungab 73b1fa7fa4 docs: correct the _perf path references after the bench moved
The `_perf/` measurement bench was gitignored and had never been committed, so
23 of the 27 scripts cited by the perf documents were already lost. The 11
surviving files (4 scripts, cfg_dm1.json, 5 result JSONs, the 53 MB MJPG clip)
have been moved to D:\orisys_ws\docs\performance\perf-measurements\ and the
scripts patched to resolve their defaults next to themselves. The directory has
been removed from this repo.

These three files said the scripts were deleted, which was only half true:
they are moved, and the four surviving ones still run. Repointed:

- src/orisys/sensor.py: the FILTER_PRESETS comment now names the real location
  (comment only; FILTER_PRESETS itself is unchanged).
- docs/filters.md and examples/filters/README.md: the same correction for the
  latency/noise measurement provenance.

No behaviour change.
2026-09-27 21:49:26 +08:00
ykcheungab d5b938738e docs: point perf references at the workspace docs tree; ignore .scratch/
orisys-sdk-perf-plan.md moved to D:\orisys_ws\docs\performance\ as part of the
2026-09-27 workspace cleanup. docs/filters.md and examples/filters/README.md now
name the new location; the _perf/ measurement scripts they also cited are deleted
(the document keeps every measured number).

Also ignore .scratch/, the new per-task scratch directory (see
D:\orisys_ws\AGENTS.md).
2026-09-27 21:14:20 +08:00
ykcheungab e2ac36269c added ruff code reivew 2026-09-27 19:21:55 +08:00
ykcheungab 07da372204 add drift script 2026-09-25 15:28:01 +08:00
ykcheungab 0e903792bd fix test sample s1.mp4 missing 2026-09-20 11:46:03 +08:00
ykcheungab 153f51df11 change sample s1_480.mp4 to s1.mp4 2026-09-20 11:39:02 +08:00
ykcheungab 2e7b65298b fixed test_signal_processing error 2026-09-20 11:36:30 +08:00
ykcheungab e2b906a640 fix preflight config path check 2026-09-20 11:20:47 +08:00
ykcheungab 45419a59c4 fix preflight version check 2026-09-20 11:13:21 +08:00
ykcheungab 05b2354d0d update to 0.5.3 2026-09-20 11:01:33 +08:00
ykcheungab 1cf78c9d71 fixed calibration data not loading due to path 2026-09-20 11:00:33 +08:00
ykcheungab 14fbfec62e remove hand.py, replaced by pipeline.py 2026-09-14 13:55:30 +08:00
ykcheungab 11962cdc34 配置没写的滤波键等于"关",不再回落到会滤波的默认值 - What: DEFAULT_FORCE_MEDIAN_WINDOW 3 -> 1、DEFAULT_FLOW_EMA_ALPHA 0.5 -> 1.0(DEFAULT_FORCE_LOWPASS_HZ 本就是 0.0),并把注释改成"配置文件没写的键就是关,而不是我们觉得合理的那一档"——原因写在注释里:否则设备会去滤一个没人要求滤的测量,而文件里没有任何线索;实测的延迟/降噪数据保留在注释中,作为 balanced/low_noise 预设的依据。docs/filters.md 第 2 节增加"缺省 = 关"这一条并说明没有 filters 段的配置什么都不滤,预设表里 balanced 的用途由"默认建议"改为"建议起步档"(它不再是默认值)。examples/filters/README.md 补一句"整段删掉等价于 off"。tests/test_filter_presets.py 新增 test_a_config_with_no_filters_block_filters_nothing:把 box bundle 的 filters 段整段删掉后构造真实 Sensor,断言 filters_state() 报 ema 1.0 / median 1 / cutoff 0.0 且 any_stage_applied 为 False。 - Why: 上一版只把"非法值"改成回落到关,但"键根本不写"仍会走 DEFAULT_* 从而把 EMA 0.5 + 中值 3 打开——这与"配置自描述、文件说什么就跑什么、出厂默认全关"矛盾,且现实里存在大量没有 filters 段的手写配置(例如 _perf/cfg_res_280x300.json),它们会在无人察觉的情况下继续滤波。缺省开是这条链上最后一个隐藏默认值。 - Verification: 删掉 filters 段后实测 filters_state() = (1.0, 1, 0.0) 且 any_stage_applied=False,出厂 box/mini/finger 三份同样为全关;pytest tests/test_filter_presets.py + tests/test_calibration.py → 33 passed;SDK 全量 336 passed / 15 skipped(唯一失败仍是需要 1200 帧而样例只有 577 帧的 longrun 测试);app 侧 studio 85 项与 shared 142 项测试、ui_check 554 项检查在升级后的 0.5.2 下全绿。 - Not in this change: 出厂 bundle 的三个显式值未动(本来就全关);_perf 的两个校验脚本不在仓库内,已就地补上显式 filters 段(balanced)以保持"开 vs 关"对照仍有意义;app 侧 wrapper/齿轮/CSV 17 列未动。 2026-09-14 10:53:33 +08:00
ykcheungab ae5bf0c208 断开连接不再因无头 OpenCV 抛错 - What: Sensor.disconnect() 里的 cv2.destroyAllWindows() 包上 try/except cv2.error:无头构建(opencv-python-headless)没有 highgui,destroyAllWindows 会抛 "The function is not implemented",此前会让 finally 里 disconnect() 的调用方(测试、采集退出路径)拿到一个与传感器无关的异常。 - Why: 跑 SDK 测试时发现 tests/test_calibration.py::test_sensor_cal_path_v2_applies_to_read_info 与 test_set_calibration_swaps_curve 在本机(cv2 4.14.0 headless)稳定失败,断言本身都过了,报错在 finally 的 disconnect() 第 1098 行;这不是滤波改动引入的,但会一直污染每次验证,且 SDK 明确支持无头/CI 使用(backend="cpu"、external-frame 模式)。 - Verification: 修前该模块 7 failed / 13 passed(含上述两个既有测试),修后 20 passed;SDK 全量 335 passed / 15 skipped(唯一失败为需要 1200 帧而 data/sample/s1_480p.mp4 只有 577 帧的 longrun 测试)。 - Not in this change: 未改 disconnect() 的其他分支;未引入 highgui 依赖探测,只在真正调用失败时忽略。 2026-09-14 10:44:13 +08:00
ykcheungab 7f8e5d0975 四个滤波预设落成机器可读的配置文件,并加上防漂移测试 - What: 新增 examples/filters/filters_{off,low_latency,balanced,low_noise}.json —— 每个都是完整可整块替换的 filters 段(五键齐全:flow_ema_alpha / force_median_window / force_lowpass_cutoff_hz / force_lowpass_order / force_lowpass_sample_rate_hz),数值即 FILTER_PRESETS 展开后的结果;examples/filters/README.md 说明这四份副本的存在与用途;新增 tests/test_filter_presets.py(12 项):预设名恰为四个、每个示例文件 == 默认块叠加该预设(改预设或改文件都会红)、每份都是含三个开关键的真实 filters 段、三个出厂 bundle(box/mini/finger)的 filters 段仍为全关、且用 external-frame 模式构造真实 Sensor 核对 filters_state() 报的也是全关(文件与设备必须一致)。 - Why: 之前的实现计划里写的是"四个 sensor_<preset>.json 整机配置",但整机配置要按传感器各复制一份、且必然随 bundle 漂移;README 里内嵌的 JSON 又无法被测试覆盖,等于把"建议用哪档滤波"这句话留在了一处会过时的散文里。改成"filters 片段 + 断言与 FILTER_PRESETS 逐值一致的测试",既不谎称是完整配置,也不会静默过时。 - Verification: pytest tests/test_filter_presets.py → 12 passed;SDK 全量 335 passed / 15 skipped(唯一失败仍是需要 1200 帧而样例只有 577 帧的 longrun 测试)。 - Not in this change: 未生成按传感器复制的整机配置;docs/filters.md 的预设表未纳入测试(只有这四个文件与代码常量对齐);app 侧未动。 2026-09-14 10:44:07 +08:00
ykcheungab 35190562e5 标定指纹改读生效值并按测量/读数分组 - What: calibration.py 新增 FILTER_MEASUREMENT_KEYS(flow_ema_alpha)与 FILTER_READOUT_KEYS(中值窗口、低通截止/阶数/采样率/rate_auto)、filters_fingerprint()(把 Sensor.filters_state() 归一化成 {measurement, readout, preset, source}:非法值丢弃、来源没写的键不出现而不是补默认、空映射返回 {} 表示"无记录")、filter_mismatches()(按组判读差异)与 _filters_from_config / _flat_filters_from(扁平旧格式与 preset 展开的兼容解析,显式键优先);pipeline_from_sensor_json() 新增 filters_state= 关键字——传了生效值就用生效值,否则读文件的 filters 段并标注 source="sensor.json.filters",返回结构化 filters 键,删除原先会凭空补出 0.5/3/10 的扁平键与 enabled 覆盖逻辑;Calibration.to_dict() 写 meta.filters(同时保留扁平键供旧读者按名取用);parse() 把 filters 读回 pipeline(此前 to_dict 写进 meta 的滤波键在读回时被整段丢弃,指纹实际是"只写不读");check() 改用 filter_mismatches 分开报告——measurement 组差异进 warnings 且措辞含 not comparable,readout 组差异进新增字段 CheckResult.notes(只记录、不拦截);build(extra_meta=…) 把 filters 与扁平滤波键折进结构化组。tests/test_calibration.py 新增 11 项测试(分组、preset 展开与显式键优先、enabled:false 全关、非法值为空而非默认、写入-读回往返、旧扁平文档读回、生效值压过文件、四个预设逐档核对)。docs/filters.md 新增第 7 节。 - Why: 用户决策 B 要求"Sensor.filters_state() 作为唯一生效值出口,calibration.py 指纹与 session.json 都读它、不读配置文件,指纹分 measurement/readout 两组";动手时确认旧实现三个问题叠加:来源错(读文件、被 filters.enabled 覆盖、文件没写就补默认 0.5/3/10),写进 meta 的指纹在读回时被丢弃,check() 因此永远比不出滤波差异——审计链是断的,不是不精确。 - Verification: SDK 全量 pytest 335 passed / 15 skipped,唯一失败 test_vnormal_longrun_no_outlier_jumps 需要 1200 帧而 data/sample/s1_480p.mp4 只有 577 帧(环境/数据,与本改动无关);external-frame 模式逐预设构造真实 Sensor,filters_fingerprint(filters_state()) 得 off(1.0,1,0.0) / low_latency(1.0,3,0.0) / balanced(0.5,3,0.0) / low_noise(0.3,3,10.0);写入→读回往返后 measurement/readout/preset 完整保留且 to_dict 幂等;旧扁平 meta 文档仍读回同一指纹;.repro/verify_cal_filters.py 50 项全绿。 - Not in this change: 未动 app(app 的 meta_from_sensor_json 只取 4 个键,不受影响);未升级 venv 内 SDK;examples/filters 的四个机器可读预设片段与 app 侧 wrapper/齿轮/CSV 17 列未动。 2026-09-14 10:43:58 +08:00
ykcheungab b956d071ea 滤波说明落到文档与 examples:三级位置/性质、四个预设、可追溯性清单 - What: 新增 docs/filters.md(json.load 不支持注释,故建议以"代码预设 + 文档 + 可复制配置"承载:三级滤波的作用位置与性质对照——flow_ema_alpha 在 compute_flow/HHD 之前、改变测量故两次录制不可比,中值与低通是读数滤波;取值规则:每级按值开关、非法值一律关、filters.enabled 仅作废弃别名、显式键优先于预设;FILTER_PRESETS 四个档位的延迟与用途表;configure()/filters_state() 用法;force_triplets() 两个 buffer 的语义与尖峰实测 10/999;可追溯性五条清单)与 examples/filters/README.md(off/low_latency/balanced/low_noise 四段可直接整块替换的 filters 配置、preset 等价写法、"从预设出发只改一个键"的写法、复制后用 filters_state() 核对生效值的片段)。 - Why: 用户提出"在 json 里加 comment 说明建议的滤波,或在 examples 里写",查证 configs.py 使用 json.load 且环境无 json5/commentjson,加注释会导致加载失败;因此把建议做成可执行预设并在文档与 examples 中给出可复制版本。 - Verification: 文档中的四个预设与 FILTER_PRESETS 逐值一致;示例片段与 sensors/*/sensor.json 的 filters 段结构一致;filters_state()/force_triplets() 的行为已用 external-frame 模式实测(预设解析结果与尖峰 10/999 对照)。 - Not in this change: 未生成四个独立的 sensor_<preset>.json 整机配置文件(README 里给的是整块 filters 段);calibration.py 指纹尚未改读 filters_state();venv 内 SDK 未升级;app 侧 wrapper/齿轮与 CSV 17 列未动。 2026-09-14 10:36:00 +08:00
ykcheungab 0beb36a1af 滤波预设与生效值出口:FILTER_PRESETS + filters_state() + force_triplets() - What: 新增 FILTER_PRESETS(off/low_latency/balanced/low_noise,数值取自 _perf 的延迟表,low_noise 含 10 Hz 低通)与 available_filter_presets() 供 UI 取用;配置可写 filters.preset 且显式键优先于预设值;新增 Sensor.filters_state()(preset + measurement 组 flow_ema_alpha/flow_ema_applied/hhd_downsample/dis_mode + readout 组中值与低通的窗口/截止/阶数/采样率/是否 auto/是否生效 + sample_rate_source + any_stage_applied + source="sensor.filters_state")作为唯一"生效值"出口;新增 Sensor.force_triplets() 一次返回 {filtered:{raw,newton}, unfiltered:{raw,newton}},并在 _apply_force_filter 中于链后、标定前保存 _force_filtered_raw(修正:此前滤波后的 raw 计数没有保存,force_triplets 的 filtered.raw 实际返回的是输入);reset() 清空未滤波缓存。 - Why: 用户要求配置自描述(A)并把"生效值"作为审计唯一依据(B),同时要保留滤波前数据以便"两个 buffer、两个单位都保留";顺带证实并修掉了 filtered.raw 指向输入这一标签错误。 - Verification: external-frame 模式逐预设构造真实 Sensor 打印 filters_state:off/low_latency/balanced/low_noise 分别为 (ema 1.0 off, median 1 off, lp 0.0 off) / (1.0 off, 3 on, 0.0 off) / (0.5 on, 3 on, 0.0 off) / (0.3 on, 3 on, 10.0 on);注入 10,10,10,999,10,10 观察力三元组,filtered.raw 在尖峰帧为 10.0、unfiltered.raw 为 999.0,证明中值生效且未滤波 buffer 完整;compileall 通过。 - Not in this change: calibration.py 指纹尚未改读 filters_state(),examples/filters 与 docs/filters.md 未加,venv 里的 SDK 未升级(app 行为不变,仍硬编码 10 Hz 低通),app 侧 wrapper/齿轮/CSV 17 列未动。 2026-09-14 10:35:51 +08:00
ykcheungab 1adb1c30f0 滤波配置自描述:去掉覆盖式总开关,每级按自己的值生效 - What: sensor.py 删除 filters.enabled 这个覆盖层——flow_ema_alpha / force_median_window / force_lowpass_cutoff_hz 现在各自按值生效(0<α<1 才滤、≤1 关中值、≤0 关低通),非法值一律回落到"关"而不再回落到"默认开"(原实现里非法取值的 except 分支会把 cutoff 变回 10 Hz);enabled 仅作废弃别名保留(显式 false = 三级全关)并在 verbose 时提示弃用,true 不再有覆盖作用;_filters_enabled 改为派生量(任一级在滤才为真)供旧调用方查询。三份 sensors/{box,mini,finger}/sensor.json 的 filters 段改为自描述且默认全关(flow_ema_alpha 1.0 / force_median_window 1 / force_lowpass_cutoff_hz 0.0 / order 2 / sample_rate "auto")。 - Why: 用户问"flow_ema_alpha 是不是受 filter_enabled 控制、这样合适吗",查证后确认三级都被同一总开关覆盖,且出厂配置写 enabled:false 却仍列着 0.5/3,导致文件说的与设备实际跑的不一致(审计/指纹会误报);同时 flow_ema_alpha 位于 HHD 之前、改变的是测量本身,与中值/低通这类读数滤波性质不同,不应被一个开关捆绑。 - Verification: 用 SDK 的 external-frame 模式逐个 bundle 构造真实 Sensor 并读取解析结果:box/mini/finger 均为 ema=1.0、median=1、cutoff=0.0、lowpass_on=False、any_stage=False;三份 JSON 解析通过;compileall 通过;SDK 内 _filters_enabled 仅剩派生定义一处引用。 - Not in this change: 未动采集路径与 read_info/force_triplets(第 2 步),未动 calibration.py 指纹(第 2 步),未升级 venv 里的 SDK(第 5 步)——因此 app 当前行为不变(它仍在运行时硬编码 lowpass_cutoff_hz=10,压过本配置),"默认关"要到 app 改用 FilterSettings.apply_to(sensor) 后才在界面体现。 2026-09-14 10:26:39 +08:00
ykcheungab 514874e2f4 added 5 sensors multi-stream support, 5 finger reach > 100 fps stable using .mp4 video 2026-09-13 12:18:27 +08:00
ykcheungab 0d5d8ff70d update filter, flow ema 0.5 + med 3 2026-09-12 19:36:36 +08:00
ykcheungab 5f78115429 fix pypi cannot check after upload 2026-09-12 17:43:40 +08:00
ykcheungab 488c8b0bc1 fixed testpypi upload 2026-09-12 17:34:46 +08:00
ykcheungab 94f38bc389 fix github release 2026-09-12 17:01:35 +08:00
ykcheungab 174bceabc4 automate publishing: release github + release pypi 2026-09-12 16:50:55 +08:00
ykcheungab 388f34ca85 faster linux build 2026-09-12 16:36:59 +08:00
ykcheungab 35af74f639 exclude docs from wheel; harden bundle item handling; single-source version 2026-09-12 16:19:34 +08:00
ykcheungab 57bef85e32 fix preflight 2026-09-12 16:00:40 +08:00
ykcheungab 9537cf91a1 added preflight check 2026-09-12 15:49:28 +08:00
zhangyijian 9c23ba3d50 Merge pull request 'Feature/cal json v2' (#21) from feature/cal_json_v2 into dev
Reviewed-on: #21
2026-09-10 21:48:55 +08:00
ykcheungab 96f4428230 added calibration data handling functions 2026-09-10 21:47:36 +08:00
ykcheungab 9cbba08d68 tuned sensitivity to max. by default 2026-09-03 15:23:51 +08:00
ykcheungab ad680b77dc 0.5.1 2026-09-02 21:47:51 +08:00
185 changed files with 19951 additions and 1200 deletions
+10 -1
View File
@@ -10,6 +10,7 @@ __pycache__/
/build/dist/ /build/dist/
/build/lib/ /build/lib/
/build/pyinstaller_work/ /build/pyinstaller_work/
/build/release/reports/
/build/*.spec /build/*.spec
/dist/ /dist/
/dist_obfuscated/ /dist_obfuscated/
@@ -23,12 +24,14 @@ __pycache__/
# Local tools / editor state # Local tools / editor state
/.cursor/ /.cursor/
/.vscode/ /.vscode/
/_perf/
# Local data / models # Local data / models
/models/ /models/
/data/* /data/*
!/data/sample/ !/data/sample/
!/data/sample/** !/data/sample/**
!/data/calibrations/
# Generated docs / tooling outputs # Generated docs / tooling outputs
/docs/.quarto/ /docs/.quarto/
@@ -41,4 +44,10 @@ __pycache__/
# Legacy / scratch folders # Legacy / scratch folders
/old_orisys/ /old_orisys/
/tools/calibration/ /tools/calibration/
# Scratch output of examples/ (multi_stream.py's default --out-dir)
/outputs/
# Agent scratch: one directory per task, see D:\orisys_ws\AGENTS.md
.scratch/
+2
View File
@@ -2,6 +2,8 @@
include pyproject.toml include pyproject.toml
include setup.py include setup.py
include README.md include README.md
# Build-time input for the PyPI long description (read from the checkout).
include docs/pypi_readme.md
recursive-include config *.json recursive-include config *.json
recursive-include sensors * recursive-include sensors *
recursive-include build/common *.py recursive-include build/common *.py
+35 -6
View File
@@ -9,7 +9,8 @@ PyPI: [https://pypi.org/project/orisys/](https://pypi.org/project/orisys/)
## Repository Layout ## Repository Layout
- `src/orisys/`: installable SDK package - `src/orisys/`: installable SDK package
- `config/`: default and lab/runtime JSON configurations - `sensors/`: product bundles containing the current `sensor.json` configurations and assets
- `config/`: legacy JSON configurations retained for compatibility
- `examples/`: runnable demos (`basic_pipeline.py`, `mini_pipeline.py`, `finger_pipeline*.py`) plus shared `gui/`, `pipeline/`, `viz/` - `examples/`: runnable demos (`basic_pipeline.py`, `mini_pipeline.py`, `finger_pipeline*.py`) plus shared `gui/`, `pipeline/`, `viz/`
- `scripts/`: maintenance, calibration, packaging, and demo scripts - `scripts/`: maintenance, calibration, packaging, and demo scripts
- `tools/`: non-package tooling such as patch-generation utilities - `tools/`: non-package tooling such as patch-generation utilities
@@ -73,12 +74,13 @@ Run commands from the repository root.
## Configuration ## Configuration
The canonical configuration files now live in the repo-level `config/` folder. Product configurations live in `sensors/<name>/sensor.json`. The repo-level `config/`
folder remains available for older configurations and aliases.
The package loader `orisys.configs` resolves configs from: The package loader `orisys.configs` resolves configs from:
1. an explicit JSON file path 1. an explicit JSON file path
2. a config file stem under `config/` (for example `box`, `mini`, `finger`) 2. `sensors/<name>/sensor.json` (for example `box`, `mini`, `finger`)
3. a product alias such as `dd02` -> `dd02-ov` 3. a file stem under the legacy `config/` folder, including aliases such as `dd02` -> `dd02-ov`
4. a backward-compatible JSON `"name"` match, such as `"prototype"` 4. a backward-compatible JSON `"name"` match, such as `"prototype"`
Examples: Examples:
@@ -90,12 +92,39 @@ sensor = orisys.Sensor(0, config_name="box")
cfg = orisys.export_config("box", "./config/exported_config.json") cfg = orisys.export_config("box", "./config/exported_config.json")
``` ```
For multiple physical sensors, keep calibration files in a user-selected directory:
`<calibration_dir>/<product>/<device_id>.json` (for example,
`data/calibrations/box/BOX-001.json`). Each file uses the existing calibration JSON
format and must contain matching `identity.sensor_type` and `identity.sn`, Newton
curves for FN/FX/FY, and a compatible measurement pipeline. Enter the device ID
manually until hardware identification is available:
The source checkout provides `data/calibrations/{box,mini,finger}/` folders; device
files placed there are ignored by Git. This location is optional: pass
`--calibration-dir data/calibrations`
and `--device-id BOX-001` to `examples/basic_pipeline.py` to select a file there.
Installed wheels do not create or search this directory automatically.
```python
sensor = orisys.Sensor(0, config_name="box", calibration_dir="data/calibrations", device_id="BOX-001")
```
If that ID has no file, the sensor reports raw force; it never falls back to the
product's default. An invalid or mismatched file raises `CalibrationError`.
`FingerRuntime.open` accepts the same `calibration_dir` and `device_id` arguments.
Existing calls without a device ID still use the product bundle default, and
explicit `cal_path` remains available for one-off use. `cal_path` and
`calibration_dir` cannot be combined. Pass `device_id` with `cal_path` to validate
that explicit file against the selected device.
## Examples and Tools ## Examples and Tools
- Example entry points: `examples/README.md` - Example entry points: `examples/README.md`
- User documentation: `docs/get_started.md` - Contributor start: [docs/get_started.en.md](docs/get_started.en.md)
- Chinese installation guide: [docs/get_started.md](docs/get_started.md)
- Build launcher: `python build/run.py` (see `build/README.md`) - Build launcher: `python build/run.py` (see `build/README.md`)
- Manylinux / PyPI binary wheel: `docs/publishing.md` - Build and publish guide: [docs/publishing.en.md](docs/publishing.en.md)
- Chinese release runbook: [docs/publishing.md](docs/publishing.md)
- Specialized utilities: `tools/` - Specialized utilities: `tools/`
## Publishing (Linux manylinux) ## Publishing (Linux manylinux)
+86 -2
View File
@@ -12,6 +12,7 @@ build/
README.md README.md
common/ shared helpers (paths, pyproject, verify) common/ shared helpers (paths, pyproject, verify)
wheel/ SDK .whl (Nuitka + manylinux Docker) wheel/ SDK .whl (Nuitka + manylinux Docker)
release/ local release gate (preflight / audit)
gui/ gui/
common/ shared PyInstaller driver common/ shared PyInstaller driver
ory_finger/ ORY-FINGER dashboard ory_finger/ ORY-FINGER dashboard
@@ -26,6 +27,7 @@ build/
```bash ```bash
python build/run.py wheel-win python build/run.py wheel-win
python build/run.py wheel-manylinux python build/run.py wheel-manylinux
python build/run.py wheel-manylinux -Architecture aarch64 -PythonVersions 3.10
# Single Version # Single Version
python build/run.py wheel python build/run.py wheel
@@ -38,9 +40,88 @@ Direct module paths (same behavior):
```bash ```bash
python build/wheel/build.py python build/wheel/build.py
build\wheel\win.bat build\wheel\win.bat
.\build\wheel\docker\run.ps1 .\build\wheel\docker\run.ps1 -SkipImageBuild # reuse the image when only sources changed
``` ```
`wheel-manylinux` defaults to `x86_64`; `-Architecture aarch64` builds ARM64 Linux
wheels in `quay.io/pypa/manylinux_2_28_aarch64`. Docker Desktop must use Linux
containers and support ARM64 emulation on an Intel/AMD host. This runs the ARM64
compiler under QEMU, so compilation can be much slower than on native ARM64.
The default Python versions remain 3.10, 3.11 and 3.12; `-PythonVersions` selects
which ABIs to compile. The image still prepares tooling for all three ABIs.
The default image names are `orisys-manylinux` and `orisys-manylinux-aarch64`.
`-ImageName` overrides the name; a reused image must match the requested
architecture. For example, reuse the ARM64 image after source-only changes:
```powershell
.\build\wheel\docker\run.ps1 -Architecture aarch64 -PythonVersions 3.10 -SkipImageBuild
```
ARM64 wheels target glibc Linux, including Q6A and Jetson systems with compatible
Python ABIs and glibc versions (2.28 or newer for this build image). They do not
target Android. The build requires binary ARM64 wheels for the runtime
dependencies and performs the existing isolated import check. Validate on each
board before delivery; CUDA, JetPack and Qualcomm GPU/NPU runtimes need separate
compatibility checks. Builds share staging directories, so run architectures
and Python ABIs sequentially.
The image bakes in per-ABI build tooling and a per-ABI wheelhouse of the runtime
deps, so the entrypoint no longer re-resolves them over the network. See
`docs/publishing.md` §6 for build-time details, why `-SkipImageBuild` matters,
and what to change before attempting to parallelize the per-ABI builds.
### Release gate
Local only: never publishes. `--check-pypi` probes the index; native Windows
venvs may download dependencies. Linux verification uses local Docker images
and their offline wheelhouses. See `docs/publishing.en.md` for the full flow.
```bash
python build/run.py release preflight --version <version>
python build/run.py release preflight --all-wheel-pythons --check-pypi
python build/run.py release audit SDK\<version>\orisys-<version>-cp312-cp312-win_amd64.whl
```
`preflight` runs: repo state → version consistency → version-string scan →
duplicate guard (fails only when the version is already released: a local git
tag, or PyPI with `--check-pypi`) → pytest → bundle precheck → per-wheel static
content/metadata assertions → isolated runtime verification in one venv per wheel
(imports + bundled assets + real frames, no camera). A populated `SDK/<version>/`
is the expected input, so build first:
```bash
python build/run.py wheel-win
python build/run.py wheel-manylinux
python build/run.py wheel-manylinux -Architecture aarch64
python build/run.py release preflight --all-wheel-pythons --check-pypi
```
Reports land in `build/release/reports/<version>/`.
### Publish
Both default to a dry run; `--apply` is required for anything irreversible.
```bash
python build/run.py release auth # credential status
python build/run.py release auth --set-token pypi # hidden prompt, writes ~/.pypirc
python build/run.py release github --version <version> # preview
python build/run.py release github --version <version> --apply # sync + tag + push + Release
python build/run.py release pypi --version <version> --apply # TestPyPI only
python build/run.py release pypi --version <version> --confirm-pypi --apply # + PyPI
python build/run.py release pypi --version <version> --pypi-only --apply # skip TestPyPI
```
Both refuse to act unless `release preflight` passed on a clean commit with no
waived options, and every wheel still hashes to what that report recorded.
Runtime evidence must cover every wheel hash. `--all-wheel-pythons` verifies all
staged platforms and ABIs, using conda for Windows and matching Docker images for
Linux. GitHub attaches all certified wheels; PyPI verifies every uploaded
Python/platform wheel by downloading it and comparing its hash.
Tokens are only ever read from a hidden prompt or stdin, never from argv.
See `docs/publishing.md` §3.
### GUI ### GUI
```bash ```bash
@@ -53,13 +134,16 @@ python build/run.py gui box # scaffold / NotImplemented
```bash ```bash
python build/run.py verify path\to\orisys.whl python build/run.py verify path\to\orisys.whl
python build/run.py verify path\to\orisys.whl --contents --assets --runtime
python build/run.py verify --gui-dist --product ory_finger python build/run.py verify --gui-dist --product ory_finger
``` ```
## Outputs ## Outputs
- Wheels: `SDK/<version>/`, `dist/repaired/` - Wheels: `SDK/<version>/`, `dist/repaired/`
- Preflight reports: `build/release/reports/<version>/`
- Finger GUI: `dist/OrisysDashboard/` - Finger GUI: `dist/OrisysDashboard/`
- Future mini/box: `dist/OryMini/`, `dist/OryBox/` - Future mini/box: `dist/OryMini/`, `dist/OryBox/`
Generated work files under `build/nuitka/` and `build/pyinstaller_work/` are gitignored. Generated work files under `build/nuitka/`, `build/release/reports/` and
`build/pyinstaller_work/` are gitignored.
+17 -2
View File
@@ -1,6 +1,21 @@
"""Post-build smoke tests for wheels and product GUI folders.""" """Post-build smoke tests for wheels and product GUI folders."""
from common.verify.gui import GUI_PRODUCTS, verify_gui_dist from common.verify.gui import GUI_PRODUCTS, verify_gui_dist
from common.verify.wheel import verify_built_wheel from common.verify.wheel import (
verify_built_wheel,
verify_wheel,
verify_wheel_assets,
verify_wheel_runtime,
)
from common.verify.wheel_contents import WheelCheckResult, inspect_wheel
__all__ = ["GUI_PRODUCTS", "verify_built_wheel", "verify_gui_dist"] __all__ = [
"GUI_PRODUCTS",
"WheelCheckResult",
"inspect_wheel",
"verify_built_wheel",
"verify_gui_dist",
"verify_wheel",
"verify_wheel_assets",
"verify_wheel_runtime",
]
+108 -1
View File
@@ -3,9 +3,27 @@
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import sys
from pathlib import Path from pathlib import Path
from common.verify import GUI_PRODUCTS, verify_built_wheel, verify_gui_dist _BUILD_ROOT = Path(__file__).resolve().parents[2]
if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT # noqa: E402
from common.verify import ( # noqa: E402
GUI_PRODUCTS,
inspect_wheel,
verify_built_wheel,
verify_gui_dist,
verify_wheel,
verify_wheel_assets,
verify_wheel_runtime,
)
from common.verify.wheel import ( # noqa: E402
DEFAULT_RUNTIME_CONFIG,
default_verification_video,
)
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
@@ -18,6 +36,31 @@ def main(argv: list[str] | None = None) -> int:
help="GUI product profile (default: ory_finger)", help="GUI product profile (default: ory_finger)",
) )
parser.add_argument("--dist-dir", type=Path, default=None, help="Override GUI dist folder") parser.add_argument("--dist-dir", type=Path, default=None, help="Override GUI dist folder")
parser.add_argument(
"--contents",
action="store_true",
help="Run static metadata/content assertions on the wheel",
)
parser.add_argument(
"--assets",
action="store_true",
help="Install the wheel in a temp venv and assert bundled assets load",
)
parser.add_argument(
"--runtime",
action="store_true",
help="Install the wheel and process real frames (no camera required)",
)
parser.add_argument("--expected-version", default=None, help="Assert the wheel version")
parser.add_argument(
"--strict-bundle",
action="store_true",
help="Treat bundled-file drift from the source tree as a failure",
)
parser.add_argument("--video", type=Path, default=None, help="Clip for --runtime")
parser.add_argument("--config", default=DEFAULT_RUNTIME_CONFIG, help="Config for --runtime")
parser.add_argument("--frames", type=int, default=40, help="Frames for --runtime")
parser.add_argument("--offline", action="store_true", help="Reuse parent-env dependencies")
parser.add_argument("wheel", nargs="?", type=Path, help="Wheel path for import smoke test") parser.add_argument("wheel", nargs="?", type=Path, help="Wheel path for import smoke test")
args = parser.parse_args(argv) args = parser.parse_args(argv)
@@ -26,6 +69,70 @@ def main(argv: list[str] | None = None) -> int:
return 0 return 0
if args.wheel is None: if args.wheel is None:
parser.error("Provide a wheel path or pass --gui-dist") parser.error("Provide a wheel path or pass --gui-dist")
exit_code = 0
if args.contents:
result = inspect_wheel(
args.wheel,
expected_version=args.expected_version,
repo_root=REPO_ROOT,
strict_bundle=args.strict_bundle,
)
print(result.render())
if not result.ok:
exit_code = 1
if args.assets and args.runtime:
video = args.video or default_verification_video(REPO_ROOT)
if video is None:
print("[FAIL] runtime: no clip found; pass --video or set ORISYS_VERIFY_VIDEO")
return 1
try:
record = verify_wheel(
args.wheel,
video=video,
config_name=args.config,
frames=args.frames,
offline=args.offline,
)
print(f"[ok ] {record['imports']}")
print(f"[ok ] {record['assets']}")
print(f"[ok ] runtime: frames={record['frames']} flow={record['flow_shape']}")
except Exception as exc:
print(f"[FAIL] isolated verification: {type(exc).__name__}: {exc}")
exit_code = 1
return exit_code
if args.assets:
try:
print(verify_wheel_assets(args.wheel))
except Exception as exc:
print(f"[FAIL] bundled assets: {type(exc).__name__}: {exc}")
exit_code = 1
if args.runtime:
video = args.video or default_verification_video(REPO_ROOT)
if video is None:
print("[FAIL] runtime: no clip found; pass --video or set ORISYS_VERIFY_VIDEO")
return 1
try:
info = verify_wheel_runtime(
args.wheel,
video=video,
config_name=args.config,
frames=args.frames,
offline=args.offline,
)
print(f"[ok ] runtime: {info}")
except Exception as exc:
print(f"[FAIL] runtime: {type(exc).__name__}: {exc}")
exit_code = 1
if args.contents or args.assets or args.runtime:
return exit_code
# Default behaviour is unchanged: import smoke test in an isolated venv.
verify_built_wheel(args.wheel) verify_built_wheel(args.wheel)
return 0 return 0
+397 -13
View File
@@ -1,11 +1,26 @@
"""Isolated venv smoke tests for packaged orisys wheels.""" """Isolated venv smoke tests for packaged orisys wheels.
Three levels of confidence, all in a throwaway virtualenv so the active
environment is never touched:
* :func:`verify_built_wheel` - the public API imports (fast, original check);
* :func:`verify_wheel_assets` - bundled configs/sensors resolve and load via
``importlib.resources``;
* :func:`verify_wheel_runtime` - the compiled wheel actually processes real
frames through the public pipeline with no camera attached.
Every level can be bootstrapped from another interpreter so each supported
CPython ABI can be verified against its own wheel.
"""
from __future__ import annotations from __future__ import annotations
import os
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
import venv import venv
from contextlib import contextmanager
from pathlib import Path from pathlib import Path
# Public SDK surface that must import from a released wheel. # Public SDK surface that must import from a released wheel.
@@ -38,6 +53,167 @@ assert Stitcher is not None
print(f"orisys {orisys.__version__} public API imports OK") print(f"orisys {orisys.__version__} public API imports OK")
""" """
# Bundled data that must be resolvable and loadable from inside site-packages.
# The wheel ships the compiled package, so these paths only exist if the bundle
# step actually ran for this build.
_ASSET_SCRIPT = r"""
import sys
from importlib.metadata import distribution
from pathlib import Path
import numpy as np
import orisys
# Locate bundled data through the installed distribution, NOT through
# orisys.__file__ / importlib.resources. Nuitka compiles the package into one
# extension module whose __init__ installs nuitka_module_loader, which reports a
# synthetic __file__ (e.g. ...\site-packages\orisys\orisys\__init__.py) and a
# synthetic __path__ ending in '\not_existing'. Neither points at the real data
# directory, so importlib.resources silently finds nothing in a Nuitka wheel.
dist = distribution("orisys")
dist_root = Path(dist.locate_file("")).resolve()
package_root = Path(dist.locate_file("orisys")).resolve()
_diagnosis = (
f"orisys distribution resolved to {dist_root} (sys.prefix={sys.prefix!r}, "
f"sys.executable={sys.executable!r}), outside the verification venv. A stale "
"orisys in the parent environment is visible through --system-site-packages; "
"run `pip uninstall -y orisys` there, or re-run without --offline."
)
assert str(Path(sys.prefix).resolve()) in str(dist_root), _diagnosis
assert package_root.is_dir(), f"package directory not found: {package_root}"
required = [
"configs/box.json",
"configs/finger.json",
"sensors/box/force_calibration.json",
"sensors/box/sensor.json",
"sensors/finger/manifest.json",
"sensors/finger/sensor.json",
"sensors/mini/sensor.json",
"sensors/finger/assets/30fps-0703/mesh.npz",
"sensors/finger/assets/30fps-0703/camera_intrinsics.npz",
"sensors/finger/assets/30fps-0703/world_to_cam.npy",
"sensors/finger/assets/visualization/mesh.npz",
"sensors/finger/assets/visualization/world_to_cam.npy",
]
missing = [name for name in required if not (package_root / name).is_file()]
assert not missing, (
f"bundled assets missing from the wheel: {missing}\n"
f"package root: {package_root}\ninstalled tree: "
f"{sorted(p.name for p in package_root.iterdir()) if package_root.is_dir() else '<missing>'}"
)
for name in required:
path = package_root / name
if name.endswith(".npz"):
data = np.load(str(path))
assert len(data.files) > 0, f"{name} is an empty npz"
elif name.endswith(".npy"):
arr = np.load(str(path))
assert arr.size > 0, f"{name} is an empty npy"
configs = orisys.list_config()
assert configs, "orisys.list_config() returned nothing"
print(f"bundled assets OK ({len(required)} files, {len(configs)} configs) "
f"from {package_root}")
"""
# End-to-end pipeline run against a video file: no camera, no GUI, no imshow.
_RUNTIME_SCRIPT = r"""
import json
import os
import sys
from importlib.metadata import distribution
from pathlib import Path
import numpy as np
os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0")
import orisys
# Provenance via the installed distribution: a Nuitka-compiled package reports a
# synthetic orisys.__file__ (see the asset check) and a stale orisys visible
# through --system-site-packages would otherwise satisfy this check.
dist = distribution("orisys")
dist_root = Path(dist.locate_file("")).resolve()
assert str(Path(sys.prefix).resolve()) in str(dist_root), (
f"the installed orisys came from {dist_root}, outside the verification venv "
f"{sys.prefix!r}; run `pip uninstall -y orisys` in the parent environment or "
"re-run without --offline."
)
video, config_name, frames = sys.argv[1], sys.argv[2], int(sys.argv[3])
sensor = orisys.Sensor(
vid_src=video,
config_name=config_name,
backend="cpu",
verbose=False,
)
processed = 0
try:
for _ in range(frames):
img = sensor.get_img()
assert img is not None, "get_img returned None"
assert isinstance(img, np.ndarray) and img.ndim in (2, 3)
sensor.compute_deformation(check_motion=False)
flow, vnormal, vshear = sensor.read_info(
sensor.info.VRAW, sensor.info.VNORMAL, sensor.info.VSHEAR
)
for name, array in (("flow", flow), ("vnormal", vnormal), ("vshear", vshear)):
assert np.isfinite(array).all(), f"{name} contains NaN/Inf"
contact = sensor.compute_contact()
assert isinstance(contact, dict) and "centroid_found" in contact
fnormal, fsx, fsy = sensor.read_info(
sensor.info.FNORMAL, sensor.info.FSHEARX, sensor.info.FSHEARY
)
assert np.isfinite([fnormal, fsx, fsy]).all(), "force output contains NaN/Inf"
processed += 1
finally:
sensor.disconnect()
assert processed == frames, f"only processed {processed}/{frames} frames"
print(json.dumps({
"version": orisys.__version__,
"frames": processed,
"flow_shape": list(flow.shape),
"video": video,
"config": config_name,
"package_root": str(Path(dist.locate_file("orisys")).resolve()),
}))
"""
# Default runtime-verification inputs. Overridable by CLI flags or environment.
DEFAULT_RUNTIME_CONFIG = os.getenv("ORISYS_VERIFY_CONFIG", "box")
# Clips tracked in the repository. ``data/sample/s1.mp4`` is the sanctioned
# verification input; the larger 720p clip is deliberately not listed, so a
# missing s1.mp4 fails loudly instead of silently verifying against it.
DEFAULT_VIDEO_CANDIDATES = (
Path("data") / "sample" / "s1.mp4",
)
def default_verification_video(repo_root: Path) -> Path | None:
"""Resolve the clip used for the end-to-end runtime check, if any."""
env_video = os.getenv("ORISYS_VERIFY_VIDEO")
repo_root = Path(repo_root)
if env_video:
candidate = Path(env_video)
return candidate if candidate.is_absolute() else repo_root / candidate
for relative in DEFAULT_VIDEO_CANDIDATES:
candidate = repo_root / relative
if candidate.is_file():
return candidate
return None
def _venv_python(venv_dir: Path) -> Path: def _venv_python(venv_dir: Path) -> Path:
if sys.platform == "win32": if sys.platform == "win32":
@@ -45,18 +221,226 @@ def _venv_python(venv_dir: Path) -> Path:
return venv_dir / "bin" / "python" return venv_dir / "bin" / "python"
def verify_built_wheel(wheel_path: Path) -> None: def _hermetic_env() -> dict:
"""Install the wheel in a temporary venv and smoke-test public imports.""" """Environment for the verification venv with PYTHONPATH removed.
wheel_path = wheel_path.resolve()
with tempfile.TemporaryDirectory(prefix="orisys-wheel-verify-") as tmp:
venv_dir = Path(tmp) / "venv"
venv.create(venv_dir, with_pip=True)
python = _venv_python(venv_dir)
subprocess.check_call( An inherited PYTHONPATH=src would let the checkout satisfy the checks
[python, "-m", "pip", "install", "-q", "--upgrade", "pip"], instead of the installed wheel, which is precisely the failure mode the
provenance assertions in the scripts exist to catch.
"""
return {key: value for key, value in os.environ.items() if key != "PYTHONPATH"}
def _create_venv(
venv_dir: Path,
*,
python_executable: Path | None = None,
system_site_packages: bool = False,
) -> Path:
"""Create a throwaway venv, optionally from a different interpreter."""
if python_executable is None:
venv.create(venv_dir, with_pip=True, system_site_packages=system_site_packages)
else:
command = [str(python_executable), "-m", "venv"]
if system_site_packages:
command.append("--system-site-packages")
command.append(str(venv_dir))
subprocess.check_call(command, env=_hermetic_env())
return _venv_python(venv_dir)
def _install_wheel(
python: Path, wheel_path: Path, *, offline: bool = False, env: dict | None = None
) -> None:
"""Install the wheel under test, guaranteeing it lands in the venv.
``--force-reinstall --no-deps`` is not optional: pip reports a same-version
requirement as "already satisfied" and skips the file, which is fatal here
because ``--offline`` also exposes the parent environment's site-packages.
Without it the venv check can silently validate a *different* orisys than
the one passed in (the scripts additionally assert provenance).
"""
if offline:
# Dependencies come from the parent environment via --system-site-packages.
commands = [
[
str(python),
"-m",
"pip",
"install",
"-q",
"--no-index",
"--no-deps",
"--force-reinstall",
str(wheel_path),
]
]
else:
commands = [
[str(python), "-m", "pip", "install", "-q", "--upgrade", "pip"],
[str(python), "-m", "pip", "install", "-q", str(wheel_path)],
[
str(python),
"-m",
"pip",
"install",
"-q",
"--force-reinstall",
"--no-deps",
str(wheel_path),
],
]
for command in commands:
subprocess.check_call(command, stdout=subprocess.DEVNULL, env=env)
class WheelVerificationError(RuntimeError):
"""A verification step failed inside the throwaway venv.
Carries the child's stdout/stderr so the real traceback is visible instead of
a bare non-zero exit code.
"""
def _run_script(
python: Path, script: str, args: list[str] | None = None, *, env: dict
) -> str:
"""Run a verification script in an already-prepared venv."""
completed = subprocess.run(
[str(python), "-c", script, *(args or [])],
capture_output=True,
text=True,
env=env,
check=False,
)
if completed.returncode != 0:
command = " ".join(str(part) for part in (args or []))
raise WheelVerificationError(
f"verification script failed in the venv (exit {completed.returncode}"
+ (f", args: {command}" if command else "")
+ ").\n--- stdout ---\n"
+ (completed.stdout.strip() or "<empty>")
+ "\n--- stderr ---\n"
+ (completed.stderr.strip() or "<empty>")
) )
subprocess.check_call( return completed.stdout.strip()
[python, "-m", "pip", "install", "-q", str(wheel_path)],
@contextmanager
def _installed_venv(
wheel_path: Path,
*,
python_executable: Path | None = None,
offline: bool = False,
prefix: str = "orisys-wheel-verify-",
):
"""Create a venv with the wheel installed, for one or more checks.
Creating a venv and installing the runtime dependencies is by far the most
expensive part of verification (tens of seconds plus a large download when
not using ``--offline``), so callers that run several checks must share one
venv instead of building one per check.
"""
wheel_path = wheel_path.resolve()
env = _hermetic_env()
with tempfile.TemporaryDirectory(prefix=prefix) as tmp:
venv_dir = Path(tmp) / "venv"
interpreter = python_executable or Path(sys.executable)
print(
f" [{wheel_path.name}] creating venv from {interpreter.name}"
+ ("" if offline else " and installing dependencies (first run downloads packages)")
+ " ...",
flush=True,
) )
subprocess.check_call([python, "-c", _SMOKE_IMPORT_SCRIPT]) python = _create_venv(
venv_dir,
python_executable=python_executable,
system_site_packages=offline,
)
_install_wheel(python, wheel_path, offline=offline, env=env)
yield python, env
def verify_built_wheel(wheel_path: Path, *, python_executable: Path | None = None) -> None:
"""Install the wheel in a temporary venv and smoke-test public imports."""
with _installed_venv(wheel_path, python_executable=python_executable) as (python, env):
_run_script(python, _SMOKE_IMPORT_SCRIPT, env=env)
def verify_wheel_assets(wheel_path: Path, *, python_executable: Path | None = None) -> str:
"""Assert the wheel carries loadable bundled configs and sensor assets."""
with _installed_venv(wheel_path, python_executable=python_executable) as (python, env):
return _run_script(python, _ASSET_SCRIPT, env=env)
def verify_wheel_runtime(
wheel_path: Path,
*,
video: Path,
config_name: str,
frames: int = 40,
python_executable: Path | None = None,
offline: bool = False,
) -> dict:
"""Run real frames through the installed wheel with no camera attached.
This is the only check that proves the compiled extension plus the bundled
sensor assets actually produce finite flow/force output end to end.
"""
with _installed_venv(
wheel_path, python_executable=python_executable, offline=offline
) as (python, env):
return _runtime_check(python, env, video=video, config_name=config_name, frames=frames)
def _runtime_check(
python: Path, env: dict, *, video: Path, config_name: str, frames: int
) -> dict:
video = Path(video).resolve()
if not video.is_file():
raise FileNotFoundError(f"verification video not found: {video}")
if frames < 1:
raise ValueError("frames must be >= 1")
print(f" processing {frames} frame(s) with config {config_name!r} ...", flush=True)
output = _run_script(
python, _RUNTIME_SCRIPT, [str(video), config_name, str(frames)], env=env
)
import json
return json.loads(output.splitlines()[-1])
def verify_wheel(
wheel_path: Path,
*,
video: Path | None = None,
config_name: str = DEFAULT_RUNTIME_CONFIG,
frames: int = 40,
python_executable: Path | None = None,
offline: bool = False,
check_assets: bool = True,
) -> dict:
"""Run every isolated-check level for one wheel inside a single venv.
Use this instead of calling the individual ``verify_wheel_*`` helpers: it
pays the venv/dependency cost once and returns the runtime record plus the
asset and import summaries.
"""
with _installed_venv(
wheel_path, python_executable=python_executable, offline=offline
) as (python, env):
imports = _run_script(python, _SMOKE_IMPORT_SCRIPT, env=env)
record: dict = {"wheel": Path(wheel_path).name, "imports": imports}
if check_assets:
record["assets"] = _run_script(python, _ASSET_SCRIPT, env=env)
else:
record["assets"] = "asset check skipped (--no-assets)"
if video is not None:
record.update(
_runtime_check(
python, env, video=video, config_name=config_name, frames=frames
)
)
return record
+502
View File
@@ -0,0 +1,502 @@
"""Static content and metadata validation for built orisys wheels.
Everything here is pure standard library: the wheel is opened as a zip and
inspected in place. No installation, no network, no hardware.
The checks encode the guarantees a Nuitka-compiled release must satisfy:
* the filename, ``METADATA`` and ``RECORD`` agree on name/version/tags;
* PyPI-visible metadata (``Requires-Python``, classifiers, project URLs,
license) is present instead of silently dropped by the hand-written
``setup.py`` template in :mod:`wheel.package`;
* no Python sources ship inside ``orisys/`` (IP containment) while the compiled
extension is present;
* every file declared by ``[[tool.orisys.bundle.items]]`` is present and
byte-identical to the source tree;
* no caches, build leftovers or scratch directories leak in.
"""
from __future__ import annotations
import base64
import hashlib
import re
import sys
import zipfile
from dataclasses import dataclass, field
from email.parser import Parser
from pathlib import Path
_BUILD_ROOT = Path(__file__).resolve().parents[1]
if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT))
from wheel.layout import resolve_bundle_source_files # noqa: E402
from wheel.package import is_compliant_manylinux_wheel # noqa: E402
from common.paths import REPO_ROOT # noqa: E402
from common.pyproject_utils import ( # noqa: E402
bundle_items,
extras_require,
install_requires,
)
# PyPI's default per-project file limit is 100 MB; keep headroom.
MAX_WHEEL_BYTES = 90 * 1024 * 1024
_FILENAME_RE = re.compile(
r"^(?P<name>[^-]+)-(?P<version>[^-]+)-(?P<python>[^-]+)-(?P<abi>[^-]+)-(?P<platform>.+)\.whl$"
)
_COMPILED_EXT_RE = re.compile(r"^orisys/__init__\..+\.(?:pyd|so)$")
_FORBIDDEN_PREFIXES = (
"test/",
"tests/",
"data/",
"build/",
"dist/",
"sdk/",
".git",
)
_FORBIDDEN_SUFFIXES = (".pyc", ".pyo", ".c", ".pdb", ".map", ".spec")
@dataclass(frozen=True)
class Check:
name: str
ok: bool
detail: str = ""
def render(self) -> str:
return f"[{'ok' if self.ok else 'FAIL'}] {self.name}: {self.detail}"
@dataclass
class WheelCheckResult:
wheel: Path
version: str | None = None
name: str | None = None
platform_tag: str | None = None
entry_count: int = 0
uncompressed_bytes: int = 0
file_bytes: int = 0
checks: list[Check] = field(default_factory=list)
warnings: list[str] = field(default_factory=list)
@property
def ok(self) -> bool:
return all(check.ok for check in self.checks)
@property
def failures(self) -> list[Check]:
return [check for check in self.checks if not check.ok]
def add(self, name: str, ok: bool, detail: str = "") -> bool:
self.checks.append(Check(name=name, ok=ok, detail=detail))
return ok
def warn(self, message: str) -> None:
self.warnings.append(message)
def render(self) -> str:
lines = [
f"wheel: {self.wheel.name}",
f" version={self.version} platform={self.platform_tag} "
f"entries={self.entry_count} file_size={self.file_bytes} "
f"uncompressed={self.uncompressed_bytes}",
]
lines.extend(f" {check.render()}" for check in self.checks)
lines.extend(f" [warn] {message}" for message in self.warnings)
lines.append(f" RESULT: {'PASS' if self.ok else 'FAIL'}")
return "\n".join(lines)
def _sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def _requirement_name(spec: str) -> str:
return re.split(r"[<>=!~\[; ]", spec, maxsplit=1)[0].strip().lower().replace("_", "-")
def _metadata_message(zf: zipfile.ZipFile, result: WheelCheckResult):
candidates = [name for name in zf.namelist() if name.endswith(".dist-info/METADATA")]
if not candidates:
result.add("metadata-present", False, "no *.dist-info/METADATA entry in the wheel")
return None
raw = zf.read(candidates[0]).decode("utf-8", errors="replace")
message = Parser().parsestr(raw)
result.add("metadata-present", True, candidates[0])
return message
def _check_filename(result: WheelCheckResult, expected_version: str | None) -> None:
match = _FILENAME_RE.match(result.wheel.name)
if match is None:
result.add("filename", False, f"unparsable wheel filename: {result.wheel.name}")
return
result.name = match.group("name")
result.version = match.group("version")
result.platform_tag = match.group("platform")
problems = []
if result.name != "orisys":
problems.append(f"distribution name is {result.name!r}, expected 'orisys'")
if expected_version is not None and result.version != expected_version:
problems.append(f"filename version {result.version!r} != expected {expected_version!r}")
result.add("filename", not problems, "; ".join(problems) or result.wheel.name)
def _check_metadata(
result: WheelCheckResult,
message,
expected_version: str | None,
repo_root: Path | None,
) -> None:
if message is None:
return
result.add(
"metadata-name",
message.get("Name") == "orisys",
f"Name={message.get('Name')!r}",
)
declared = message.get("Version")
expected = expected_version or result.version
if expected is not None and declared != expected:
result.add(
"metadata-version",
False,
f"METADATA Version={declared!r} != expected {expected!r}",
)
else:
result.add("metadata-version", True, f"Version={declared!r}")
requires_python = (message.get("Requires-Python") or "").strip()
result.add(
"metadata-requires-python",
bool(requires_python),
f"Requires-Python={requires_python!r}" if requires_python else "Requires-Python is missing",
)
classifiers = message.get_all("Classifier") or []
result.add(
"metadata-classifiers",
bool(classifiers),
f"{len(classifiers)} classifier(s)" if classifiers else "no Classifier fields",
)
project_urls = message.get_all("Project-URL") or []
result.add(
"metadata-project-urls",
bool(project_urls),
f"{len(project_urls)} Project-URL(s)" if project_urls else "no Project-URL fields",
)
license_values = [
(message.get("License") or "").strip(),
(message.get("License-Expression") or "").strip(),
]
has_license = any(license_values) or any(
value.startswith("License ::") for value in classifiers
)
result.add(
"metadata-license",
has_license,
f"License={license_values[0]!r} License-Expression={license_values[1]!r}"
if has_license
else "no License / License-Expression / 'License ::' classifier",
)
dynamic = message.get_all("Dynamic") or []
result.add(
"metadata-no-dynamic-version",
"version" not in dynamic,
f"Dynamic={dynamic!r}" if dynamic else "no Dynamic fields",
)
if dynamic:
result.warn(f"METADATA still declares Dynamic fields: {dynamic!r}")
if repo_root is not None and (repo_root / "pyproject.toml").is_file():
requires_dist = message.get_all("Requires-Dist") or []
provided = {_requirement_name(entry) for entry in requires_dist}
missing_core = [
spec for spec in install_requires(repo_root) if _requirement_name(spec) not in provided
]
result.add(
"metadata-requires-dist",
not missing_core,
f"missing core Requires-Dist entries: {missing_core!r}"
if missing_core
else f"{len(requires_dist)} Requires-Dist entries",
)
extras = message.get_all("Provides-Extra") or []
missing_extras = sorted(set(extras_require(repo_root)) - set(extras))
result.add(
"metadata-extras",
not missing_extras,
f"missing Provides-Extra: {missing_extras!r}"
if missing_extras
else f"Provides-Extra={sorted(extras)!r}",
)
def _check_compiled_only(result: WheelCheckResult, names: list[str]) -> None:
compiled = [name for name in names if _COMPILED_EXT_RE.match(name)]
result.add(
"compiled-extension",
bool(compiled),
f"{compiled[0]}" if compiled else "no orisys/__init__.<abi>.pyd|.so entry",
)
# orisys/examples/** is shipped as package data on purpose; every other
# orisys/**/*.py means source code leaked past the Nuitka compile step.
sources = [
name
for name in names
if name.startswith("orisys/")
and name.endswith(".py")
and not name.startswith("orisys/examples/")
]
result.add(
"no-python-sources",
not sources,
f"python sources leaked into the wheel: {sources[:5]!r}"
if sources
else "compiled package ships no python sources (examples excluded by design)",
)
blend = [name for name in names if name.startswith("orisys_blend/")]
result.add(
"orisys-blend-present",
"orisys_blend/__init__.py" in blend and "orisys_blend/kernel.py" in blend,
f"{sorted(blend)!r}",
)
def _check_bundle_fidelity(
result: WheelCheckResult,
zf: zipfile.ZipFile,
names: set[str],
repo_root: Path | None,
strict: bool,
) -> None:
if repo_root is None or not (repo_root / "pyproject.toml").is_file():
result.warn("pyproject.toml not found; bundle fidelity checks skipped")
return
missing: list[str] = []
mismatched: list[str] = []
checked = 0
for item in bundle_items(repo_root):
target = str(item["target"]).replace("\\", "/")
source_dir = repo_root / item["source"]
for source_file in resolve_bundle_source_files(repo_root, item):
if item.get("preserve_tree", False):
relative = source_file.relative_to(source_dir).as_posix()
entry = f"orisys/{target}/{relative}"
else:
entry = f"orisys/{target}/{source_file.name}"
if entry not in names:
missing.append(entry)
continue
checked += 1
if _sha256(zf.read(entry)) != _sha256(source_file.read_bytes()):
mismatched.append(entry)
# Both entries here mean "the wheel does not match this revision of the
# source tree". Against a fresh build that is fatal; against an already
# released wheel it is expected, so ``strict`` (set by preflight, unset by
# audit) decides whether it fails or warns.
if missing:
detail = (
f"{len(missing)} declared file(s) absent from the wheel (added to the "
f"source tree after this build?): {missing[:5]!r}"
)
result.add("bundle-completeness", not strict, detail)
if not strict:
result.warn(detail)
else:
result.add(
"bundle-completeness", True, f"{checked} declared bundled file(s) present"
)
if mismatched:
detail = (
f"{len(mismatched)} of {checked} bundled file(s) differ from the source "
f"tree (wheel predates or postdates it): {mismatched[:5]!r}"
)
result.add("bundle-fidelity", not strict, detail)
if not strict:
result.warn(detail)
else:
result.add(
"bundle-fidelity",
True,
f"{checked} bundled file(s) byte-identical to source",
)
def _check_forbidden_entries(result: WheelCheckResult, names: list[str]) -> None:
offenders = []
for name in names:
lowered = name.lower()
if lowered.startswith(_FORBIDDEN_PREFIXES) or lowered.endswith(_FORBIDDEN_SUFFIXES) or "__pycache__" in lowered or ".egg-info" in lowered:
offenders.append(name)
result.add(
"no-forbidden-entries",
not offenders,
f"unexpected entries: {offenders[:5]!r}" if offenders else "no caches/leftovers/scratch paths",
)
def _check_archive_paths(result: WheelCheckResult, names: list[str]) -> None:
unsafe = [
name
for name in names
if name.startswith(("/", "\\")) or ".." in Path(name.replace("\\", "/")).parts
]
result.add(
"archive-paths",
not unsafe,
f"absolute or traversing entries: {unsafe[:5]!r}" if unsafe else "all entries are relative",
)
def _check_record(result: WheelCheckResult, zf: zipfile.ZipFile, names: list[str]) -> None:
records = [name for name in names if name.endswith(".dist-info/RECORD")]
if not records:
result.add("record", False, "no *.dist-info/RECORD entry")
return
bad_paths: list[str] = []
bad_hashes: list[str] = []
bad_sizes: list[str] = []
present = set(names)
for line in zf.read(records[0]).decode("utf-8").splitlines():
if not line.strip():
continue
parts = line.split(",")
while len(parts) < 3:
parts.append("")
path, hash_field, size_field = parts[0], parts[1], parts[2]
if path not in present:
bad_paths.append(path)
continue
if not hash_field.startswith("sha256="):
continue
expected = hash_field[len("sha256=") :]
actual = base64.urlsafe_b64encode(
hashlib.sha256(zf.read(path)).digest()
).rstrip(b"=").decode("ascii")
if actual != expected:
bad_hashes.append(path)
if size_field and size_field.isdigit():
if int(size_field) != len(zf.read(path)):
bad_sizes.append(path)
problems = []
if bad_paths:
problems.append(f"paths not in archive: {bad_paths[:5]!r}")
if bad_hashes:
problems.append(f"sha256 mismatch: {bad_hashes[:5]!r}")
if bad_sizes:
problems.append(f"size mismatch: {bad_sizes[:5]!r}")
result.add(
"record",
not problems,
"; ".join(problems) if problems else f"{records[0]} verifies",
)
def _check_platform_tag(result: WheelCheckResult) -> None:
tag = result.platform_tag
if tag is None:
return
if "win" in tag:
ok = tag == "win_amd64"
result.add("platform-tag", ok, f"{tag!r} (expected 'win_amd64')")
return
ok = is_compliant_manylinux_wheel(result.wheel)
result.add(
"platform-tag",
ok,
f"{tag!r}" if ok else f"{tag!r} is not a supported manylinux x86_64 or aarch64 tag",
)
def _check_size(result: WheelCheckResult, max_bytes: int) -> None:
result.add(
"size-limit",
result.file_bytes <= max_bytes,
f"{result.file_bytes} bytes (limit {max_bytes})",
)
def inspect_wheel(
wheel_path: Path,
*,
expected_version: str | None = None,
repo_root: Path | None = None,
max_bytes: int = MAX_WHEEL_BYTES,
strict_bundle: bool = True,
) -> WheelCheckResult:
"""Validate one built wheel without installing it.
``strict_bundle`` controls how the wheel's bundled files are compared with the
current source tree: ``True`` (gate a fresh build, used by
``release preflight``) fails on any missing or changed declared file;
``False`` (audit an already-released wheel) reports the same findings as
warnings.
"""
wheel_path = Path(wheel_path).resolve()
if repo_root is None:
repo_root = REPO_ROOT if (REPO_ROOT / "pyproject.toml").is_file() else None
else:
repo_root = Path(repo_root).resolve()
result = WheelCheckResult(wheel=wheel_path)
if not wheel_path.is_file():
result.add("wheel-exists", False, f"not a file: {wheel_path}")
return result
result.file_bytes = wheel_path.stat().st_size
result.add("wheel-exists", True, str(wheel_path))
_check_filename(result, expected_version)
_check_platform_tag(result)
_check_size(result, max_bytes)
try:
zf = zipfile.ZipFile(wheel_path)
except zipfile.BadZipFile as exc:
result.add("zip-integrity", False, f"not a valid zip: {exc}")
return result
with zf:
corrupt = zf.testzip()
result.add(
"zip-integrity",
corrupt is None,
"CRC check passed" if corrupt is None else f"corrupt entry: {corrupt}",
)
names = zf.namelist()
name_set = set(names)
result.entry_count = len(names)
result.uncompressed_bytes = sum(info.file_size for info in zf.infolist())
_check_archive_paths(result, names)
_check_forbidden_entries(result, names)
message = _metadata_message(zf, result)
_check_metadata(result, message, expected_version, repo_root)
_check_compiled_only(result, names)
_check_bundle_fidelity(result, zf, name_set, repo_root, strict_bundle)
_check_record(result, zf, names)
return result
+1 -1
View File
@@ -6,9 +6,9 @@ import os
import shutil import shutil
import subprocess import subprocess
import sys import sys
from collections.abc import Callable
from dataclasses import dataclass, field from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
from typing import Callable
_BUILD_ROOT = Path(__file__).resolve().parents[2] _BUILD_ROOT = Path(__file__).resolve().parents[2]
if str(_BUILD_ROOT) not in sys.path: if str(_BUILD_ROOT) not in sys.path:
+1
View File
@@ -10,6 +10,7 @@ if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT)) sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT from common.paths import REPO_ROOT
from gui.common.pyinstaller_driver import GuiBuildConfig from gui.common.pyinstaller_driver import GuiBuildConfig
ENTRY_SCRIPT = REPO_ROOT / "examples" / "basic_pipeline.py" ENTRY_SCRIPT = REPO_ROOT / "examples" / "basic_pipeline.py"
+1
View File
@@ -15,6 +15,7 @@ if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT)) sys.path.insert(0, str(_BUILD_ROOT))
from common.verify.gui import verify_gui_dist from common.verify.gui import verify_gui_dist
from gui.common.pyinstaller_driver import build_with_pyinstaller from gui.common.pyinstaller_driver import build_with_pyinstaller
from gui.ory_finger.config import make_config from gui.ory_finger.config import make_config
+1
View File
@@ -10,6 +10,7 @@ if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT)) sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT from common.paths import REPO_ROOT
from gui.common.pyinstaller_driver import GuiBuildConfig from gui.common.pyinstaller_driver import GuiBuildConfig
ENTRY_SCRIPT = REPO_ROOT / "examples" / "finger_pipeline_3dview.py" ENTRY_SCRIPT = REPO_ROOT / "examples" / "finger_pipeline_3dview.py"
+1
View File
@@ -10,6 +10,7 @@ if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT)) sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT from common.paths import REPO_ROOT
from gui.common.pyinstaller_driver import GuiBuildConfig from gui.common.pyinstaller_driver import GuiBuildConfig
ENTRY_SCRIPT = REPO_ROOT / "examples" / "mini_pipeline.py" ENTRY_SCRIPT = REPO_ROOT / "examples" / "mini_pipeline.py"
+7
View File
@@ -0,0 +1,7 @@
"""Local release tooling: preflight gate and per-wheel audit.
Entry point is ``python build/run.py release <command>``; see
:mod:`release.preflight` for the gate steps and :mod:`release.cli` for the CLI.
Imports are intentionally not re-exported here so importing a submodule never
triggers the whole subsystem.
"""
+398
View File
@@ -0,0 +1,398 @@
"""PyPI credential setup: locate, inspect and write ``~/.pypirc``.
Editing the config from a command rather than by hand avoids the two mistakes
that actually happen: putting something other than ``__token__`` in ``username``,
and pasting a TestPyPI token into the ``[pypi]`` section (the two indexes reject
each other's tokens with 403, and both token kinds start with ``pypi-``).
The writer does targeted text edits rather than rewriting through configparser,
so the explanatory comments in the file survive.
"""
from __future__ import annotations
import configparser
import getpass
import os
import sys
from dataclasses import dataclass, field
from pathlib import Path
INDEXES: dict[str, dict[str, str | None]] = {
"pypi": {
"upload_url": "https://upload.pypi.org/legacy/",
"json_url": "https://pypi.org/pypi/orisys/json",
"simple_url": "https://pypi.org/simple/",
"token_url": "https://pypi.org/manage/account/token/",
"register_url": None,
},
"testpypi": {
"upload_url": "https://test.pypi.org/legacy/",
"json_url": "https://test.pypi.org/pypi/orisys/json",
"simple_url": "https://test.pypi.org/simple/",
"token_url": "https://test.pypi.org/manage/account/token/",
"register_url": "https://test.pypi.org/account/register/",
},
}
# Substrings that mark a token slot that was never filled in.
PLACEHOLDER_MARKERS = ("PASTE", "REPLACE", "TOKEN-HERE", "YOUR-TOKEN", "XXXX")
DEFAULT_TEMPLATE = """\
# PyPI / TestPyPI credentials for twine.
#
# Keep this file out of any git repository: a token committed once stays in the
# history even after you delete it.
#
# * username is literally "__token__" - never an account name or email
# * PyPI and TestPyPI tokens both start with "pypi-" but are NOT
# interchangeable: each index rejects the other's with 403
# * TestPyPI is a separate service with a separate account
[distutils]
index-servers =
"""
@dataclass
class CredentialState:
index: str
configured: bool
source: str | None = None
problem: str | None = None
@dataclass
class TwineLookup:
"""What twine's own resolver says about one index."""
available: bool
known: bool
config: dict = field(default_factory=dict)
error: str | None = None
def pypirc_path() -> Path:
"""Where twine looks by default: ``~/.pypirc`` (same expansion twine uses)."""
return Path(os.path.expanduser("~")) / ".pypirc"
def looks_like_placeholder(token: str) -> bool:
upper = token.upper()
return any(marker in upper for marker in PLACEHOLDER_MARKERS)
def parse_pypirc(text: str) -> dict[str, dict[str, str]]:
"""Section -> key -> value, tolerating comments and continuation lines."""
sections: dict[str, dict[str, str]] = {}
current: str | None = None
for raw in text.splitlines():
line = raw.strip()
if not line or line.startswith(("#", ";", "%")):
continue
if line.startswith("[") and line.endswith("]"):
current = line[1:-1].strip()
sections.setdefault(current, {})
continue
if current is not None and "=" in line:
key, _, value = line.partition("=")
sections[current][key.strip()] = value.strip()
return sections
def credential_state(
index: str,
*,
path: Path | None = None,
env: dict | None = None,
) -> CredentialState:
"""Is ``index`` usable by twine, and why not when it is not.
Delegates to ``twine.utils.get_config`` when twine is importable, because the
only definition that matters is twine's. The important subtlety that a naive
check misses: **twine ignores any section not listed in
``[distutils] index-servers``** (with a fallback of ``pypi testpypi`` when the
key is absent). A ``[testpypi]`` section without that listing looks configured
but twine reports "Missing 'testpypi' section".
"""
if index not in INDEXES:
raise ValueError(f"unknown index {index!r}; expected one of {sorted(INDEXES)}")
env = os.environ if env is None else env
if env.get("TWINE_PASSWORD"):
return CredentialState(index, True, "environment variables")
path = path or pypirc_path()
if not path.is_file():
return CredentialState(index, False, None, f"{path} does not exist yet")
try:
text = path.read_text(encoding="utf-8", errors="ignore")
except OSError as exc:
return CredentialState(index, False, None, f"cannot read {path}: {exc}")
sections = parse_pypirc(text)
if index not in sections and index not in index_servers(text):
# Do not blame index-servers for a section that simply is not there.
return CredentialState(index, False, None, f"{path} has no [{index}] section")
lookup = _twine_lookup(index, path)
if lookup.error:
return CredentialState(index, False, None, f"twine cannot read {path}: {lookup.error}")
if lookup.available:
if not lookup.known:
return CredentialState(index, False, None, _ignored_section(index, path, text, sections))
username = str(lookup.config.get("username") or "").strip()
password = str(lookup.config.get("password") or "").strip()
if not password:
# twine knows the index but resolved no password for it: almost always
# a section that index-servers does not list.
detail = _ignored_section(index, path, text, sections)
return CredentialState(index, False, None, detail)
else:
# twine is not installed: replicate its rule locally.
if index not in index_servers(text):
return CredentialState(index, False, None, _ignored_section(index, path, text, sections))
section = sections[index]
username = section.get("username", "").strip()
password = section.get("password", "").strip()
if looks_like_placeholder(password):
return CredentialState(
index, False, str(path), f"[{index}] still contains a placeholder token"
)
if username != "__token__":
return CredentialState(
index,
False,
str(path),
f"[{index}] username is {username!r}; API tokens need the literal '__token__'",
)
return CredentialState(index, True, str(path))
def _ignored_section(
index: str, path: Path, text: str, sections: dict[str, dict[str, str]]
) -> str:
"""Name the real cause: absent section, or present but not listed by twine."""
if index not in sections:
return f"{path} has no [{index}] section"
if index not in index_servers(text):
return (
f"[{index}] exists in {path} but is not listed in "
"[distutils] index-servers, so twine ignores it entirely; "
"fix with `release auth --repair`"
)
return f"twine resolves no password for [{index}] from {path}"
def _twine_lookup(index: str, path: Path) -> TwineLookup:
"""Ask twine itself; ``available=False`` when twine is not installed.
``get_config`` is not enough on its own: it always contains a ``pypi`` entry
(and a ``testpypi`` one when listed) even when it never copied any
credentials, so the resolved username/password are what matter.
"""
try:
from twine import utils
except ImportError:
return TwineLookup(available=False, known=False)
try:
config = utils.get_config(str(path))
except Exception as exc: # twine reports malformed files as exceptions
return TwineLookup(available=True, known=False, error=f"{type(exc).__name__}: {exc}")
section = config.get(index)
if section is None:
return TwineLookup(available=True, known=False)
return TwineLookup(available=True, known=True, config=dict(section))
def index_servers(text: str) -> set[str]:
"""Names twine honours from ``[distutils] index-servers``.
Replicates ``twine.utils.get_config``, including its surprising asymmetry:
* key **absent** -> falls back to ``pypi testpypi``
* key **present but empty** -> an empty list, so *no* section is honoured
(the fallback does not apply), and every section looks configured while
twine silently ignores it
* continuation lines are part of the value, so ``index-servers =`` followed
by indented names is the normal form
"""
parser = configparser.RawConfigParser()
try:
parser.read_string(text)
except configparser.Error:
return set()
raw = parser.get("distutils", "index-servers", fallback="pypi testpypi")
return set(raw.split())
def validate_token(token: str) -> list[str]:
"""Problems with a pasted token. Empty and whitespace are fatal; the rest warn."""
return [problem for problem in _token_problems(token) if problem.startswith("ERROR")]
def token_warnings(token: str) -> list[str]:
return [problem for problem in _token_problems(token) if problem.startswith("WARN")]
def _token_problems(token: str) -> list[str]:
problems: list[str] = []
if not token:
problems.append("ERROR the token is empty")
return problems
if any(char.isspace() for char in token):
problems.append("ERROR the token contains whitespace (copy/paste artefact?)")
if not token.startswith("pypi-"):
problems.append(
"WARN PyPI API tokens start with 'pypi-'; a legacy account password also "
"works but needs username != '__token__'"
)
if len(token) < 20:
problems.append("WARN the token looks too short; check the paste was complete")
return problems
def unlisted_sections(path: Path | None = None) -> list[str]:
"""Known indexes whose section exists but is missing from ``index-servers``."""
path = path or pypirc_path()
if not path.is_file():
return []
text = path.read_text(encoding="utf-8", errors="ignore")
present = set(parse_pypirc(text))
listed = index_servers(text)
return [index for index in sorted(INDEXES) if index in present and index not in listed]
def repair_index_servers(path: Path | None = None) -> list[str]:
"""List sections that exist but are not in ``index-servers``.
This is the state a hand-edited file lands in: the section is present, so it
looks configured, but twine ignores it completely. Returns the indexes added.
"""
path = path or pypirc_path()
if not path.is_file():
return []
text = path.read_text(encoding="utf-8")
present = set(parse_pypirc(text))
added: list[str] = []
for index in sorted(INDEXES):
if index in present and index not in index_servers(text):
text = ensure_index_server(text, index)
added.append(index)
if added:
path.write_text(text, encoding="utf-8", newline="")
_restrict_permissions(path)
return added
def upsert_token(text: str, index: str, token: str) -> str:
"""Set ``password`` inside ``[index]``, creating the section if needed."""
lines = text.splitlines()
start = next((i for i, line in enumerate(lines) if line.strip() == f"[{index}]"), None)
if start is None:
if lines and lines[-1].strip():
lines.append("")
lines.extend(
[
f"[{index}]",
f"repository = {INDEXES[index]['upload_url']}",
"username = __token__",
f"password = {token}",
]
)
return "\n".join(lines) + "\n"
end = len(lines)
for offset in range(start + 1, len(lines)):
stripped = lines[offset].strip()
if stripped.startswith("[") and stripped.endswith("]"):
end = offset
break
insert_at = end
for offset in range(start + 1, end):
stripped = lines[offset].strip()
if stripped.startswith("password"):
lines[offset] = f"password = {token}"
return "\n".join(lines) + "\n"
if stripped.startswith(("username", "repository")):
insert_at = offset + 1
lines.insert(insert_at, f"password = {token}")
return "\n".join(lines) + "\n"
def ensure_index_server(text: str, index: str) -> str:
"""Make sure ``[distutils] index-servers`` lists ``index``."""
if index in index_servers(text):
return text
lines = text.splitlines()
if "[distutils]" not in (line.strip() for line in lines):
block = ["[distutils]", "index-servers =", f" {index}", ""]
return "\n".join(block + lines) + "\n"
start = next(i for i, line in enumerate(lines) if line.strip() == "[distutils]")
for offset in range(start + 1, len(lines)):
if lines[offset].strip().startswith("index-servers"):
lines.insert(offset + 1, f" {index}")
return "\n".join(lines) + "\n"
lines.insert(start + 1, f" {index}")
return "\n".join(lines) + "\n"
def _restrict_permissions(path: Path) -> None:
"""Best effort: owner-only. Windows ACLs need icacls, not chmod."""
try:
os.chmod(path, 0o600)
except OSError:
pass
def write_token(index: str, token: str, *, path: Path | None = None) -> Path:
"""Write ``token`` into ``[index]`` of the pypirc file, preserving comments."""
if index not in INDEXES:
raise ValueError(f"unknown index {index!r}; expected one of {sorted(INDEXES)}")
path = path or pypirc_path()
if path.is_file():
text = path.read_text(encoding="utf-8")
else:
text = DEFAULT_TEMPLATE
text = upsert_token(text, index, token)
text = ensure_index_server(text, index)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8", newline="")
_restrict_permissions(path)
return path
def prompt_for_token(index: str, *, prompt_fn=None, out=None) -> str | None:
"""Ask for a token with hidden input. Returns ``None`` if input is unavailable."""
out = out or sys.stdout
meta = INDEXES[index]
print(f"PyPI API token for [{index}]", file=out)
print(f" create one at: {meta['token_url']}", file=out)
if meta.get("register_url"):
print(f" needs an account first: {meta['register_url']}", file=out)
print(" input is hidden and is written only to your pypirc file", file=out)
ask = prompt_fn or getpass.getpass
try:
return ask("token: ").strip()
except (EOFError, KeyboardInterrupt, OSError):
# OSError covers environments where stdin cannot be read at all (pytest
# capture, a detached CI stdin): treat that as "cannot prompt", never as a
# crash.
print("", file=out)
print("aborted: no token entered", file=out)
return None
def read_token_from_stdin() -> str:
"""For CI: read the token from stdin instead of a prompt or the command line."""
return (sys.stdin.readline() or "").strip()
+421
View File
@@ -0,0 +1,421 @@
"""Command line for ``python build/run.py release``.
Commands: ``preflight`` (local gate), ``audit`` (single wheel), ``auth`` (PyPI
credentials in ~/.pypirc), ``github`` (sync the public repo + publish a GitHub
Release), ``pypi`` (TestPyPI then PyPI). The publish commands are dry-run unless
``--apply`` is passed, and tokens are never accepted as command-line arguments.
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
_BUILD_ROOT = Path(__file__).resolve().parents[1]
if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT # noqa: E402
from common.verify.wheel import ( # noqa: E402
DEFAULT_RUNTIME_CONFIG,
default_verification_video,
verify_wheel,
verify_wheel_assets,
verify_wheel_runtime,
)
from common.verify.wheel_contents import inspect_wheel # noqa: E402
from release import auth, github, pypi # noqa: E402
from release.preflight import PreflightOptions, run_preflight # noqa: E402
def _cmd_preflight(argv: list[str]) -> int:
parser = argparse.ArgumentParser(
prog="build/run.py release preflight",
description="Gate a release locally: version, tests, wheel content, runtime.",
)
parser.add_argument("--version", help="release version (default: pyproject's dynamic version)")
parser.add_argument("--wheel-dir", type=Path, help="directory to inspect (default: SDK/<version>)")
parser.add_argument("--video", type=Path, help="clip for the end-to-end runtime check")
parser.add_argument("--config", default=DEFAULT_RUNTIME_CONFIG, help="config name for that check")
parser.add_argument("--frames", type=int, default=40, help="frames to process (default: 40)")
parser.add_argument("--skip-tests", action="store_true", help="skip pytest (recorded in the report)")
parser.add_argument("--longrun", action="store_true", help="include the slow long-run stability test")
parser.add_argument("--allow-dirty", action="store_true", help="allow a dirty working tree")
parser.add_argument(
"--force",
action="store_true",
help="accept a version that a git tag already marks as released",
)
parser.add_argument("--check-pypi", action="store_true", help="read PyPI and fail if the version exists")
parser.add_argument(
"--all-wheel-pythons",
action="store_true",
help="verify every staged wheel (Windows conda environments and Linux Docker images)",
)
parser.add_argument("--offline", action="store_true", help="reuse parent-env deps instead of PyPI")
parser.add_argument("--no-runtime", action="store_true", help="skip the isolated runtime check")
parser.add_argument("--no-assets", action="store_true", help="skip the bundled-asset check")
parser.add_argument("--keep-going", action="store_true", help="collect every failure instead of stopping")
parser.add_argument("--report-dir", type=Path, help="override build/release/reports")
args = parser.parse_args(argv)
options = PreflightOptions(
version=args.version,
wheel_dir=args.wheel_dir,
video=args.video,
config_name=args.config,
frames=args.frames,
skip_tests=args.skip_tests,
longrun=args.longrun,
allow_dirty=args.allow_dirty,
force=args.force,
check_pypi=args.check_pypi,
all_wheel_pythons=args.all_wheel_pythons,
offline=args.offline,
keep_going=args.keep_going,
report_dir=args.report_dir,
no_runtime=args.no_runtime,
no_assets=args.no_assets,
)
return run_preflight(options, repo_root=REPO_ROOT)
def _cmd_audit(argv: list[str]) -> int:
parser = argparse.ArgumentParser(
prog="build/run.py release audit",
description="Inspect one built wheel without a release context.",
)
parser.add_argument("wheel", type=Path, help="path to an .whl")
parser.add_argument("--expected-version", help="assert the wheel reports this version")
parser.add_argument(
"--strict-bundle",
action="store_true",
help="treat bundled-file drift from the current source tree as a failure",
)
parser.add_argument("--runtime", action="store_true", help="also install and run the wheel")
parser.add_argument("--assets", action="store_true", help="also assert bundled assets load")
parser.add_argument("--video", type=Path, help="clip for --runtime")
parser.add_argument("--config", default=DEFAULT_RUNTIME_CONFIG)
parser.add_argument("--frames", type=int, default=40)
parser.add_argument("--offline", action="store_true")
parser.add_argument(
"--python",
type=Path,
default=None,
help="interpreter to build the verification venv from (default: the running one)",
)
args = parser.parse_args(argv)
wheel = args.wheel
if not wheel.is_file():
print(f"[ERROR] wheel not found: {wheel}", file=sys.stderr)
return 2
exit_code = 0
result = inspect_wheel(
wheel,
expected_version=args.expected_version,
repo_root=REPO_ROOT,
strict_bundle=args.strict_bundle,
)
print(result.render())
if not result.ok:
exit_code = 1
if args.assets and args.runtime:
# One venv for both checks: building it (and downloading dependencies in
# online mode) is the expensive part.
video = args.video or default_verification_video(REPO_ROOT)
if video is None:
print("[FAIL] runtime: no verification clip; pass --video or set ORISYS_VERIFY_VIDEO")
return 1
try:
record = verify_wheel(
wheel,
video=video,
config_name=args.config,
frames=args.frames,
python_executable=args.python,
offline=args.offline,
)
print(f"[ok ] {record['imports']}")
print(f"[ok ] {record['assets']}")
print(f"[ok ] runtime: frames={record['frames']} flow={record['flow_shape']}")
except Exception as exc:
print(f"[FAIL] isolated verification: {type(exc).__name__}: {exc}")
exit_code = 1
return exit_code
if args.assets:
try:
print(verify_wheel_assets(wheel, python_executable=args.python))
except Exception as exc:
print(f"[FAIL] bundled assets: {type(exc).__name__}: {exc}")
exit_code = 1
if args.runtime:
video = args.video or default_verification_video(REPO_ROOT)
if video is None:
print("[FAIL] runtime: no verification clip; pass --video or set ORISYS_VERIFY_VIDEO")
return 1
try:
info = verify_wheel_runtime(
wheel,
video=video,
config_name=args.config,
frames=args.frames,
python_executable=args.python,
offline=args.offline,
)
print(f"[ok ] runtime: {info}")
except Exception as exc:
print(f"[FAIL] runtime: {type(exc).__name__}: {exc}")
exit_code = 1
return exit_code
def _cmd_github(argv: list[str]) -> int:
parser = argparse.ArgumentParser(
prog="build/run.py release github",
description="Sync orisys-sdk-release, tag, and publish a GitHub Release.",
)
parser.add_argument(
"--version", required=True, help="release version, e.g. <major>.<minor>.<patch>"
)
parser.add_argument(
"--apply",
action="store_true",
help="actually mutate the release repo, push, and create the release",
)
parser.add_argument("--notes-file", type=Path, default=None, help="release notes markdown")
parser.add_argument(
"--attach-preflight",
action="store_true",
help="also upload the preflight report as a release asset (contains local paths)",
)
parser.add_argument(
"--allow-dirty-release",
action="store_true",
help="proceed even if the release repo has uncommitted changes",
)
parser.add_argument("--skip-fetch", action="store_true", help="skip `git fetch --tags`")
parser.add_argument(
"--skip-checks", action="store_true", help="skip the manifest's downstream checks"
)
parser.add_argument(
"--skip-release-upload",
action="store_true",
help="stop after pushing the tag and print the `gh release create` command",
)
args = parser.parse_args(argv)
return github.publish(
github.Options(
version=args.version,
apply=args.apply,
notes_file=args.notes_file,
attach_preflight=args.attach_preflight,
allow_dirty_release=args.allow_dirty_release,
skip_fetch=args.skip_fetch,
skip_checks=args.skip_checks,
skip_release_upload=args.skip_release_upload,
)
)
def _cmd_auth(argv: list[str]) -> int:
parser = argparse.ArgumentParser(
prog="build/run.py release auth",
description=(
"Inspect or set up PyPI credentials in ~/.pypirc. "
"Tokens are read from a hidden prompt or stdin, never from argv."
),
)
parser.add_argument(
"--set-token",
choices=sorted(auth.INDEXES),
default=None,
help="store a token for this index (prompts unless --token-stdin)",
)
parser.add_argument(
"--token-stdin",
action="store_true",
help="read the token from stdin instead of prompting (for scripts/CI)",
)
parser.add_argument(
"--repair",
action="store_true",
help=(
"add existing sections to [distutils] index-servers; twine ignores "
"sections that are not listed there"
),
)
parser.add_argument(
"--index",
choices=sorted(auth.INDEXES),
default="pypi",
help="index to report on when not setting a token (default: pypi)",
)
args = parser.parse_args(argv)
if args.repair:
path = auth.pypirc_path()
added = auth.repair_index_servers(path)
if added:
print(f"[ok ] added {added} to [distutils] index-servers in {path}")
else:
print(f"[ok ] nothing to repair in {path}")
for index in sorted(auth.INDEXES):
state = auth.credential_state(index)
flag = "[ok ]" if state.configured else "[-- ]"
detail = state.source if state.configured else state.problem
print(f"{flag} [{index}] {detail}")
return 0 if auth.credential_state(args.index).configured else 1
if args.set_token:
if args.token_stdin:
token = auth.read_token_from_stdin()
else:
token = auth.prompt_for_token(args.set_token)
if not token:
print("[FAIL] no token entered")
return 1
for problem in auth.validate_token(token):
print(f"[FAIL] {problem.removeprefix('ERROR ')}")
return 1
for warning in auth.token_warnings(token):
print(f"[warn] {warning.removeprefix('WARN ')}")
path = auth.write_token(args.set_token, token)
print(f"[ok ] wrote [{args.set_token}] credentials to {path}")
state = auth.credential_state(args.set_token)
if not state.configured:
print(f"[FAIL] the file still does not validate: {state.problem}")
return 1
print(f"[ok ] [{args.set_token}] credentials verified ({state.source})")
return 0
path = auth.pypirc_path()
print(f"config file: {path} ({'exists' if path.is_file() else 'missing'})")
for index in sorted(auth.INDEXES):
state = auth.credential_state(index)
if state.configured:
print(f"[ok ] [{index}] configured ({state.source})")
else:
print(f"[-- ] [{index}] not usable: {state.problem}")
unlisted = auth.unlisted_sections(path)
if unlisted:
print()
print(
f"[warn] {unlisted} have sections that [distutils] index-servers does not "
"list; twine ignores those entirely"
)
print(" fix with: python build/run.py release auth --repair")
print()
print("set a token with:")
for index in sorted(auth.INDEXES):
if not auth.credential_state(index).configured:
print(f" python build/run.py release auth --set-token {index}")
print(" (TestPyPI needs its own account; if you only hold a production token,")
print(" skip it and publish with `release pypi --pypi-only`)")
return 0 if auth.credential_state(args.index).configured else 1
def _cmd_pypi(argv: list[str]) -> int:
parser = argparse.ArgumentParser(
prog="build/run.py release pypi",
description="Upload the certified wheels to TestPyPI and then PyPI.",
)
parser.add_argument(
"--version", required=True, help="release version, e.g. <major>.<minor>.<patch>"
)
parser.add_argument(
"--apply",
action="store_true",
help="actually upload (uploads are permanent and cannot be overwritten)",
)
parser.add_argument(
"--confirm-pypi",
action="store_true",
help="after TestPyPI, also publish to the real PyPI",
)
parser.add_argument(
"--pypi-only",
action="store_true",
help=(
"upload straight to PyPI, skipping the TestPyPI rehearsal "
"(for when you only hold a production token)"
),
)
parser.add_argument(
"--prompt-credentials",
action="store_true",
help="if a token is missing, prompt for it and store it in ~/.pypirc",
)
parser.add_argument(
"--token-stdin",
action="store_true",
help="read a missing token from stdin instead of prompting (scripts/CI)",
)
parser.add_argument(
"--skip-install-verify",
action="store_true",
help="skip fetching the artifact back out of the index",
)
parser.add_argument("--skip-check", action="store_true", help="skip `twine check`")
args = parser.parse_args(argv)
if args.pypi_only and args.confirm_pypi:
parser.error("--pypi-only and --confirm-pypi are mutually exclusive")
if args.token_stdin and not args.prompt_credentials:
parser.error("--token-stdin requires --prompt-credentials")
token = auth.read_token_from_stdin() if args.token_stdin else None
return pypi.publish(
pypi.Options(
version=args.version,
apply=args.apply,
confirm_pypi=args.confirm_pypi,
verify_install=not args.skip_install_verify,
skip_check=args.skip_check,
pypi_only=args.pypi_only,
prompt_credentials=args.prompt_credentials,
token=token,
)
)
def main(argv: list[str] | None = None) -> int:
argv = list(sys.argv[1:] if argv is None else argv)
if not argv or argv[0] in ("-h", "--help"):
print(__doc__)
print("Commands: preflight | audit | auth | github | pypi")
return 0
command, rest = argv[0], argv[1:]
if command == "preflight":
return _cmd_preflight(rest)
if command == "audit":
return _cmd_audit(rest)
if command == "auth":
return _cmd_auth(rest)
if command == "github":
return _cmd_github(rest)
if command == "pypi":
return _cmd_pypi(rest)
print(f"Unknown release command: {command!r}", file=sys.stderr)
print("Choose from: preflight, audit, auth, github, pypi", file=sys.stderr)
return 2
if __name__ == "__main__":
raise SystemExit(main())
+446
View File
@@ -0,0 +1,446 @@
"""Sync the public release repo and publish a GitHub Release.
``python build/run.py release github`` runs, in order:
1. certified-wheels gate (a passing ``release preflight`` report must exist and
every wheel must still hash to what it recorded)
2. refuse if the version is already tagged or already has a GitHub Release
3. sync the open-source subset of this checkout into ``orisys-sdk-release``
(``tools/sync_downstreams.py`` + the manifest's ``checks``)
4. stage ``SDK/<version>/*.whl`` into ``orisys-sdk-release/release/<version>/``
5. commit, tag ``v<version>``, push branch + tag
6. create the GitHub Release with the wheels attached, then verify the assets
Nothing mutates anything unless ``--apply`` is passed: the default is a dry run
that prints every command it would run.
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
from dataclasses import dataclass, field
from pathlib import Path
_BUILD_ROOT = Path(__file__).resolve().parents[1]
if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT # noqa: E402
from release.publish import GateResult, PublishGateError, load_gate # noqa: E402
from release.runtime import PLATFORM_LABELS, wheel_target # noqa: E402
MANIFEST_PATH = REPO_ROOT / "tools" / "downstream_sync_manifest.json"
SYNC_TOOL = REPO_ROOT / "tools" / "sync_downstreams.py"
RELEASE_TARGET = "orisys-sdk-release"
@dataclass
class Options:
version: str
apply: bool = False
notes_file: Path | None = None
attach_preflight: bool = False
allow_dirty_release: bool = False
skip_fetch: bool = False
skip_checks: bool = False
skip_release_upload: bool = False
@dataclass
class CommandStep:
description: str
argv: list[str]
cwd: Path | None = None
@dataclass
class Plan:
release_repo: Path
branch: str
tag: str
assets: list[Path] = field(default_factory=list)
steps: list[CommandStep] = field(default_factory=list)
def manifest_entry() -> dict:
"""The manifest entry for the public release repo."""
payload = json.loads(MANIFEST_PATH.read_text(encoding="utf-8"))
for item in payload.get("downstreams", []):
if item.get("name") == RELEASE_TARGET:
return item
raise PublishGateError(f"{MANIFEST_PATH} has no downstream named {RELEASE_TARGET!r}")
def release_repo_path() -> Path:
return (REPO_ROOT / manifest_entry()["path"]).resolve()
def _git(repo: Path, *args: str, check: bool = True) -> str:
completed = subprocess.run(
["git", "-C", str(repo), *args],
capture_output=True,
text=True,
check=False,
)
if check and completed.returncode != 0:
raise PublishGateError(
f"git {' '.join(args)} failed in {repo}:\n{completed.stderr.strip()}"
)
return (completed.stdout or "").strip()
def build_release_notes(version: str, wheels: list[Path], git_commit: str | None) -> str:
platforms = sorted({PLATFORM_LABELS[wheel_target(wheel)[1]] for wheel in wheels})
lines = [
f"orisys {version}",
"",
f"Wheels ({', '.join(platforms)}; Python ABI is shown in each filename):",
]
lines.extend(f"- `{wheel.name}`" for wheel in wheels)
lines.append("")
if git_commit:
lines.append(
f"Built from `{git_commit[:12]}` and certified by `release preflight`."
)
return "\n".join(lines) + "\n"
def build_github_steps(
*,
version: str,
branch: str,
tag: str,
assets: list[Path],
notes_file: Path | None,
create_tag: bool = True,
title: str | None = None,
) -> list[CommandStep]:
"""The mutating commands, as pure data so they can be asserted in tests.
``create_tag=False`` is the resume path: the tag already exists (a previous
run pushed it before failing), so only the remaining steps are emitted.
"""
release_argv = [
"gh",
"release",
"create",
tag,
"--verify-tag",
"--title",
title or f"v{version}",
"--latest",
]
if notes_file is not None:
release_argv += ["--notes-file", str(notes_file)]
release_argv += [str(asset) for asset in assets]
steps: list[CommandStep] = []
if create_tag:
steps.append(CommandStep("create tag", ["git", "tag", tag]))
steps.append(CommandStep("push branch", ["git", "push", "origin", branch]))
steps.append(CommandStep("push tag", ["git", "push", "origin", tag]))
steps.append(CommandStep("create GitHub Release with assets", release_argv))
return steps
def non_github_remote_problem(repo: Path) -> str | None:
"""Why ``gh`` cannot target this repo, or ``None`` if the remote is fine.
``gh`` resolves its repository from the working directory. Running it from
this checkout instead of the release repo is a real failure mode: the SDK
repo's origin is an internal host, and ``gh`` reports the confusing "none of
the git remotes ... point to a known GitHub host".
"""
remote = _git(repo, "remote", "get-url", "origin", check=False)
if "github.com" in remote:
return None
return (
f"the release repo's origin is {remote!r}, which is not a GitHub host. "
"`gh` resolves the repository from the working directory, so point origin "
"at github.com before publishing a release"
)
def _check_gh(repo: Path) -> list[str]:
"""Problems that would stop the release step, reported before any mutation."""
problems: list[str] = []
if shutil.which("gh") is None:
problems.append(
"the `gh` CLI is not on PATH. Install it (`winget install --id GitHub.cli`) "
"and authenticate (`gh auth login`), or use --skip-release-upload"
)
return problems
remote_problem = non_github_remote_problem(repo)
if remote_problem:
problems.append(remote_problem)
completed = subprocess.run(
["gh", "auth", "status"], capture_output=True, text=True, check=False
)
if completed.returncode != 0:
problems.append("`gh auth status` failed; run `gh auth login` first")
return problems
def _existing_release_tags(repo: Path, version: str) -> list[str]:
wanted = {version, f"v{version}"}
tags = set(_git(repo, "tag", "--list").splitlines())
return sorted(wanted & tags)
def _release_exists(repo: Path, tag: str) -> bool | None:
"""Is there already a GitHub Release for ``tag``?
Returns ``None`` when that cannot be determined (``gh`` unavailable), so the
caller can distinguish "not published yet" from "unknown" instead of treating
a missing tool as proof that nothing was published.
Must run with ``cwd`` set to the release repo: `gh` resolves the target
repository from the working directory, and this checkout's own remote is an
internal host.
"""
if shutil.which("gh") is None:
return None
completed = subprocess.run(
["gh", "release", "view", tag, "--json", "tagName"],
capture_output=True,
text=True,
check=False,
cwd=str(repo),
)
return completed.returncode == 0
def _sync(*, run_checks: bool) -> None:
"""Copy the public subset into the release repo and run its checks."""
python = sys.executable
for command in (
[python, str(SYNC_TOOL), "apply", "--target", RELEASE_TARGET],
*(
[[python, str(SYNC_TOOL), "run-checks", "--target", RELEASE_TARGET]]
if run_checks
else []
),
):
completed = subprocess.run(command, cwd=REPO_ROOT, check=False)
if completed.returncode != 0:
raise PublishGateError(
f"`{' '.join(command[1:])}` failed with exit {completed.returncode}"
)
def _stage_wheels(wheels: list[Path], asset_dir: Path) -> None:
asset_dir.mkdir(parents=True, exist_ok=True)
for wheel in wheels:
shutil.copy2(wheel, asset_dir / wheel.name)
def publish(options: Options) -> int:
version = options.version
mode = "APPLY" if options.apply else "DRY RUN"
print(f"=== release github: orisys {version} ({mode}) ===")
# --- 1. gate -------------------------------------------------------
try:
gate: GateResult = load_gate(REPO_ROOT, version)
except PublishGateError as exc:
print(f"[FAIL] {exc}")
return 1
print(f"[ok ] preflight report: {gate.report_path}")
print(f" certified commit: {gate.git_commit}")
for wheel in gate.wheels:
print(f" [ok] {wheel.name} sha256={wheel.sha256[:12]} {wheel.size} bytes")
for warning in gate.warnings:
print(f"[warn] {warning}")
# --- 2. target repo ------------------------------------------------
try:
repo = release_repo_path()
except PublishGateError as exc:
print(f"[FAIL] {exc}")
return 1
if not (repo / ".git").exists():
print(f"[FAIL] {repo} is not a git repository")
return 1
try:
if not options.skip_fetch:
_git(repo, "fetch", "--tags", "--quiet")
print(f"[ok ] git fetch --tags in {repo}")
branch = _git(repo, "rev-parse", "--abbrev-ref", "HEAD")
dirty = _git(repo, "status", "--porcelain")
except PublishGateError as exc:
print(f"[FAIL] {exc}")
return 1
if dirty and not options.allow_dirty_release:
preview = "\n".join(dirty.splitlines()[:10])
print(
f"[FAIL] {repo} has uncommitted changes; commit or stash them first "
f"(or pass --allow-dirty-release):\n{preview}"
)
return 1
print(f"[ok ] release repo: {repo} ({branch}) clean={not dirty}")
tag = f"v{version}"
existing = _existing_release_tags(repo, version)
# A tag that exists without a GitHub Release means a previous run got as far
# as pushing the tag and then failed (or the upload was left manual). That is
# exactly the resumable state, so continue instead of refusing.
create_tag = not existing
if existing:
print(
f"[warn] tag(s) {existing} already exist in {repo}; skipping tag creation "
"and continuing to the release upload"
)
release_state = _release_exists(repo, tag)
if release_state is True:
print(f"[FAIL] GitHub Release {tag} already exists; nothing to publish")
return 1
if release_state is None:
print(
"[warn] cannot tell whether a GitHub Release for "
f"{tag} already exists (`gh` not installed); --apply will stop at the gh check"
)
if create_tag:
print(f"[ok ] tag {tag} does not exist in {repo}")
elif create_tag:
print(f"[ok ] no existing tag or GitHub Release for {tag}")
else:
print(f"[ok ] no GitHub Release for {tag} yet")
# A dry run is a preview of what would happen, so missing tooling is a warning
# there and a hard stop only when we are about to mutate something.
if options.skip_release_upload:
print("[warn] --skip-release-upload: the GitHub Release will be left to you")
else:
gh_problems = _check_gh(repo)
if gh_problems:
for problem in gh_problems:
print(f"[{'FAIL' if options.apply else 'warn'}] {problem}")
if options.apply:
return 1
else:
print("[ok ] gh CLI is installed and authenticated")
# --- 3. plan the mutating steps ------------------------------------
wheels = [REPO_ROOT / "SDK" / version / record.name for record in gate.wheels]
missing = [str(wheel) for wheel in wheels if not wheel.is_file()]
if missing:
print(f"[FAIL] wheels missing from disk: {missing}")
return 1
asset_dir = repo / "release" / version
assets = [asset_dir / wheel.name for wheel in wheels]
notes_file = options.notes_file or gate.report_path.parent / "github-release-notes.md"
generated_notes = build_release_notes(version, wheels, gate.git_commit) if options.notes_file is None else None
steps = build_github_steps(
version=version,
branch=branch,
tag=tag,
assets=assets,
notes_file=notes_file,
create_tag=create_tag,
)
print()
print(f"[plan] sync {len(manifest_entry().get('sync', []))} entry(ies) into {repo}")
print(f"[plan] stage {len(wheels)} wheel(s) into {asset_dir}")
if create_tag:
print(f"[plan] commit {repo} and create tag {tag}")
else:
print(f"[plan] commit {repo} (tag {tag} already exists)")
for step in steps:
manual = options.skip_release_upload and step.description.startswith(
"create GitHub Release"
)
prefix = "[plan] (left manual)" if manual else "[plan]"
print(f"{prefix} {step.description}: {' '.join(step.argv)}")
print()
if not options.apply:
if generated_notes:
print(generated_notes)
print("dry run: nothing was changed. Re-run with --apply to execute.")
return 0
# --- 4. execute ----------------------------------------------------
try:
if generated_notes:
notes_file.write_text(generated_notes, encoding="utf-8")
commit_before = _git(repo, "rev-parse", "HEAD")
_sync(run_checks=not options.skip_checks)
_stage_wheels(wheels, asset_dir)
if options.attach_preflight:
shutil.copy2(gate.report_path, asset_dir / f"{version}-preflight.json")
_git(repo, "add", "-A")
pending = _git(repo, "status", "--porcelain")
if pending:
_git(repo, "commit", "-m", f"release {version}")
print(f"[ok ] committed {len(pending.splitlines())} change(s) in {repo}")
else:
print("[ok ] nothing new to commit in the release repo")
if create_tag:
_git(repo, "tag", tag)
# Pushing an already-up-to-date ref is a no-op, so resuming is safe.
_git(repo, "push", "origin", branch)
_git(repo, "push", "origin", tag)
print(f"[ok ] pushed {branch} and {tag}")
except PublishGateError as exc:
print(f"[FAIL] {exc}")
return 1
release_argv = next(
step.argv for step in steps if step.description.startswith("create GitHub Release")
)
if options.skip_release_upload:
print()
print("[todo] publish the release yourself:")
print(" " + " ".join(release_argv))
print()
print("=== release github: repo synced and tagged; release upload left manual ===")
return 0
# `gh` resolves its target repository from the working directory, so it must
# run inside the release repo - this checkout's own origin is an internal host.
completed = subprocess.run(release_argv, check=False, cwd=str(repo))
if completed.returncode != 0:
print(f"[FAIL] gh release create exited {completed.returncode}")
return 1
print(f"[ok ] GitHub Release {tag} created")
# --- 5. verify the published assets --------------------------------
completed = subprocess.run(
["gh", "release", "view", tag, "--json", "assets,tagName"],
capture_output=True,
text=True,
check=False,
cwd=str(repo),
)
if completed.returncode != 0:
print("[FAIL] could not read back the release to verify assets")
return 1
payload = json.loads(completed.stdout or "{}")
published = {asset["name"] for asset in payload.get("assets", [])}
expected = {asset.name for asset in assets}
if options.attach_preflight:
expected.add(f"{version}-preflight.json")
if published != expected:
print(
f"[FAIL] release asset mismatch:\n expected {sorted(expected)}\n"
f" published {sorted(published)}"
)
return 1
print(f"[ok ] verified {len(published)} asset(s) on the release")
print()
print(f"=== release github PASSED (commit {commit_before[:12]} -> {_git(repo, 'rev-parse', 'HEAD')[:12]}) ===")
return 0
+580
View File
@@ -0,0 +1,580 @@
"""Local release gate: everything that must hold before publishing a wheel.
``release preflight`` answers one question with an exit code: is this checkout,
this version and these wheels safe to publish? It never publishes anything;
native verification environments may download dependencies from PyPI.
Steps, in order:
1. ``repo-state`` - git repository, clean working tree, recorded commit
2. ``version-consistency`` - __version__ == pyproject dynamic attr == wheel names
3. ``version-strings`` - no stale version literal left in tracked prose
4. ``duplicate-guard`` - already-released version (git tag, optionally PyPI)
5. ``tests`` - pytest (slow/hardware markers excluded)
6. ``bundle-precheck`` - every bundle item resolves; required assets present
7. ``wheel-inspection`` - static metadata/content assertions per wheel
8. ``wheel-runtime`` - one isolated venv per wheel: bundled assets load and
real frames flow through the public pipeline
Output lands in ``build/release/reports/<version>/preflight.{json,txt}``, and is
the contract the publish commands (``release github`` / ``release pypi``) consume:
they refuse to ship anything this run did not certify.
"""
from __future__ import annotations
import hashlib
import json
import platform
import subprocess
import sys
import time
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
_BUILD_ROOT = Path(__file__).resolve().parents[1]
if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT # noqa: E402
from common.pyproject_utils import bundle_items, get_version # noqa: E402
from common.verify.wheel import ( # noqa: E402
DEFAULT_RUNTIME_CONFIG,
default_verification_video,
verify_wheel,
)
from common.verify.wheel_contents import inspect_wheel # noqa: E402
from wheel.layout import ( # noqa: E402
resolve_bundle_source_files,
verify_required_assets,
)
from release import report as report_mod # noqa: E402
from release import version as version_mod # noqa: E402
from release.runtime import verify_manylinux_wheel, wheel_target # noqa: E402
@dataclass
class Step:
name: str
ok: bool
detail: str = ""
duration_s: float = 0.0
data: dict = field(default_factory=dict)
@dataclass
class PreflightOptions:
version: str | None = None
wheel_dir: Path | None = None
video: Path | None = None
config_name: str = DEFAULT_RUNTIME_CONFIG
frames: int = 40
skip_tests: bool = False
longrun: bool = False
allow_dirty: bool = False
force: bool = False
check_pypi: bool = False
all_wheel_pythons: bool = False
offline: bool = False
keep_going: bool = False
report_dir: Path | None = None
no_runtime: bool = False
no_assets: bool = False
def _sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with Path(path).open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def _now() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
class Preflight:
def __init__(self, options: PreflightOptions, repo_root: Path | None = None) -> None:
self.options = options
self.repo_root = Path(repo_root or REPO_ROOT)
self.steps: list[Step] = []
self.git: dict = {"commit": None, "branch": None, "dirty": None}
self.wheels: list[Path] = []
self.wheel_dir: Path | None = None
self.runtime_records: list[dict] = []
self.aborted = False
self.ok = False
if options.version:
self.version = options.version
self.version_source = "explicit --version"
elif (self.repo_root / "pyproject.toml").is_file():
self.version = get_version(self.repo_root)
self.version_source = "pyproject.toml dynamic attr"
else:
# Not version-shaped on purpose: this is a sentinel for "unresolved",
# and it must not look like a release version to the version-string scan.
self.version = "unknown"
self.version_source = "unresolved"
self.report_dir = report_mod.reports_root(self.repo_root, options.report_dir) / self.version
# -- helpers ---------------------------------------------------------
def _git(self, *args: str) -> str:
try:
completed = subprocess.run(
["git", *args],
cwd=self.repo_root,
capture_output=True,
text=True,
check=False,
)
except OSError:
return ""
return (completed.stdout or "").strip()
def _execute(self, step_fn) -> None:
started = time.perf_counter()
try:
step = step_fn()
except Exception as exc: # a crashed step is a failed step, never a silent pass
step = Step(name=getattr(step_fn, "__name__", "step").lstrip("_").replace("_", "-"),
ok=False, detail=f"crashed: {type(exc).__name__}: {exc}")
step.duration_s = round(time.perf_counter() - started, 3)
self.steps.append(step)
flag = "ok " if step.ok else "FAIL"
print(f"[{flag}] {step.name} ({step.duration_s:.2f}s)")
for line in str(step.detail).splitlines():
print(f" {line}")
if not step.ok and not self.options.keep_going:
self.aborted = True
print(" stopping: pass --keep-going to collect every failure")
# -- steps -----------------------------------------------------------
def _repo_state(self) -> Step:
if not (self.repo_root / ".git").exists():
return Step("repo-state", False, f"{self.repo_root} is not a git repository")
commit = self._git("rev-parse", "HEAD")
branch = self._git("rev-parse", "--abbrev-ref", "HEAD")
porcelain = self._git("status", "--porcelain")
dirty = bool(porcelain)
self.git = {"commit": commit, "branch": branch, "dirty": dirty}
if dirty and not self.options.allow_dirty:
preview = "\n".join(porcelain.splitlines()[:20])
return Step(
"repo-state",
False,
f"working tree is dirty; commit, stash or pass --allow-dirty:\n{preview}",
data=self.git,
)
detail = f"{branch} @ {commit[:12]} dirty={dirty} version={self.version} ({self.version_source})"
if dirty:
detail += " (accepted via --allow-dirty)"
return Step("repo-state", True, detail, data=self.git)
def _version_consistency(self) -> Step:
problems: list[str] = []
data: dict = {"version": self.version, "source": self.version_source}
attr = version_mod.dynamic_version_attr(self.repo_root)
data["dynamic_attr"] = attr
if attr is None:
problems.append(
"pyproject.toml no longer declares tool.setuptools.dynamic.version.attr"
)
if (self.repo_root / "pyproject.toml").is_file():
resolved = get_version(self.repo_root)
data["resolved_version"] = resolved
if resolved != self.version:
problems.append(
f"src/orisys/__init__.py __version__ is {resolved!r} but the release "
f"version is {self.version!r}"
)
sdk_dirs = version_mod.sdk_versions(self.repo_root)
data["sdk_versions"] = sdk_dirs
for wheel in version_mod.sdk_wheels(self.repo_root, self.version):
found = version_mod.wheel_version(wheel)
if found != self.version:
problems.append(f"{wheel.name} carries version {found!r}, expected {self.version!r}")
detail = (
f"version={self.version} ({self.version_source}) dynamic_attr={attr}\n"
f"SDK/ versions present: {sdk_dirs or 'none'}"
)
if problems:
detail += "\n" + "\n".join(problems)
return Step("version-consistency", not problems, detail, data=data)
def _duplicate_guard(self) -> Step:
"""Fail only when this version has already been *released*.
A populated ``SDK/<version>/`` is the normal input to preflight (it is
what gets verified), so it is reported, not blocked. What must not happen
silently is re-releasing a version that is already out: that is detected
offline through git tags and, opt-in, against PyPI.
"""
problems: list[str] = []
lines: list[str] = []
existing = version_mod.sdk_wheels(self.repo_root, self.version)
if existing:
lines.append(
f"SDK/{self.version}/ holds {len(existing)} staged wheel(s) to verify: "
f"{[w.name for w in existing]}"
)
else:
lines.append(
f"SDK/{self.version}/ is empty; nothing is staged yet "
"(build with `python build/run.py wheel-win` / `manylinux`)"
)
tags = version_mod.git_tags(self.repo_root)
tag_hits = version_mod.released_tags_for(self.version, tags)
if tag_hits:
message = f"git tag(s) {tag_hits} already mark {self.version} as released"
if self.options.force:
lines.append(message + " (accepted via --force)")
else:
problems.append(
message + "; pass --force to re-verify or intentionally re-release"
)
else:
lines.append(
f"no local git tag marks {self.version} as released "
f"({len(tags)} tag(s) checked in this repo and {version_mod.RELEASE_REPO_DIRNAME})"
)
if self.options.check_pypi:
versions, error = version_mod.pypi_released_versions()
if error:
problems.append(f"PyPI lookup failed: {error}")
elif versions is None:
problems.append("PyPI lookup returned no version list")
elif self.version in versions:
problems.append(
f"orisys {self.version} is already published on PyPI; a PyPI upload "
"cannot be undone. Published: " + ", ".join(versions)
)
else:
lines.append(
f"PyPI has no {self.version}; published: " + (", ".join(versions) or "none")
)
else:
lines.append("PyPI existence check skipped (pass --check-pypi to enable)")
detail = "\n".join(lines)
if problems:
detail += "\n" + "\n".join(problems)
return Step("duplicate-guard", not problems, detail)
def _version_strings(self) -> Step:
"""Keep the version out of prose: one source, everything else derived.
Scans tracked text for version-looking literals that disagree with the
release version. Dependency constraints are ignored by design; deliberate
exceptions live in ``version.VERSION_STRING_ALLOWLIST``.
"""
findings = version_mod.find_version_literals(self.repo_root, self.version)
if not findings:
scanned = len(version_mod.tracked_text_files(self.repo_root))
return Step(
"version-strings",
True,
f"no stale version literal in {scanned} scanned file(s); "
f"version {self.version} lives only where it is derived from",
)
by_file: dict[str, list[str]] = {}
for path, lineno, literal in findings:
by_file.setdefault(path, []).append(f"{path}:{lineno} -> {literal}")
detail = (
f"{len(findings)} version literal(s) disagree with {self.version}. "
"Update them, or make the file version-agnostic, or add a documented "
"entry to build/release/version.py::VERSION_STRING_ALLOWLIST:\n"
+ "\n".join(line for lines in by_file.values() for line in lines)
)
return Step("version-strings", False, detail, data={"count": len(findings)})
def _tests(self) -> Step:
if self.options.skip_tests:
return Step(
"tests",
True,
"WARNING: test gate SKIPPED via --skip-tests; this run cannot certify a release",
data={"status": "skipped"},
)
marker = "not hardware" if self.options.longrun else "not slow and not hardware"
self.report_dir.mkdir(parents=True, exist_ok=True)
junit = self.report_dir / "junit.xml"
command = [
sys.executable,
"-m",
"pytest",
"-q",
"-m",
marker,
f"--junitxml={junit}",
]
try:
completed = subprocess.run(
command, cwd=self.repo_root, capture_output=True, text=True, check=False
)
except FileNotFoundError as exc:
return Step("tests", False, f"cannot run pytest: {exc}")
ok = completed.returncode == 0
tail = ((completed.stdout or "") + (completed.stderr or "")).strip().splitlines()[-12:]
detail = f"pytest -m {marker!r} in {sys.executable}\nexit code {completed.returncode}"
if tail:
detail += "\n" + "\n".join(tail)
return Step("tests", ok, detail, data={"marker": marker, "junit": str(junit)})
def _bundle_precheck(self) -> Step:
if not (self.repo_root / "pyproject.toml").is_file():
return Step("bundle-precheck", False, "pyproject.toml not found")
items = bundle_items(self.repo_root)
problems: list[str] = []
counts: dict[str, int] = {}
for item in items:
files = resolve_bundle_source_files(self.repo_root, item)
counts[str(item.get("name"))] = len(files)
if not files:
problems.append(
f"bundle item {item.get('name')!r} resolved 0 files; its patterns match nothing"
)
try:
verify_required_assets(self.repo_root, items)
except FileNotFoundError as exc:
problems.append(str(exc))
detail = "resolved files per bundle item: " + json.dumps(counts)
if problems:
detail += "\n" + "\n".join(problems)
return Step("bundle-precheck", not problems, detail, data={"counts": counts})
def _discover_wheels(self) -> str | None:
"""Populate self.wheels; return an error message when nothing was found."""
directory = (
Path(self.options.wheel_dir)
if self.options.wheel_dir
else self.repo_root / "SDK" / self.version
)
self.wheel_dir = directory
if not directory.is_dir():
return (
f"{directory} does not exist. Build first "
"(`python build/run.py wheel`) or pass --wheel-dir."
)
self.wheels = sorted(directory.glob("*.whl"))
if not self.wheels:
return f"no *.whl found in {directory}. Build first or pass --wheel-dir."
return None
def _wheel_inspection(self) -> Step:
problem = self._discover_wheels()
if problem:
return Step("wheel-inspection", False, problem)
problems: list[str] = []
blocks: list[str] = []
for wheel in self.wheels:
result = inspect_wheel(
wheel,
expected_version=self.version,
repo_root=self.repo_root,
strict_bundle=True,
)
blocks.append(result.render())
if not result.ok:
problems.append(
f"{wheel.name}: " + ", ".join(check.name for check in result.failures)
)
detail = f"{len(self.wheels)} wheel(s) in {self.wheel_dir}\n" + "\n".join(blocks)
if problems:
detail += "\nfailed checks:\n" + "\n".join(problems)
return Step("wheel-inspection", not problems, detail, data={"count": len(self.wheels)})
def _platform_token(self) -> str:
return "win_amd64" if sys.platform == "win32" else platform.machine().lower()
def _runtime_targets(self) -> list[tuple[str, Path, Path | None]]:
token = self._platform_token()
if self.options.all_wheel_pythons:
targets: list[tuple[str, Path, Path | None]] = []
for wheel in self.wheels:
tag, target = wheel_target(wheel)
executable = None
if target == "win_amd64":
if sys.platform != "win32":
raise RuntimeError("Verify Windows wheels from a Windows host")
executable = version_mod.python_for_tag(tag)
if target == "win_amd64" and executable is None:
raise RuntimeError(
f"cannot resolve an interpreter for {tag}; create the conda env "
f"py{tag[2:]} or set CONDA_ENVS_ROOT"
)
targets.append((tag, wheel, executable))
return targets
tag = f"cp{sys.version_info.major}{sys.version_info.minor}"
match = next((w for w in self.wheels if wheel_target(w) == (tag, token)), None)
if match is None:
raise RuntimeError(
f"no {tag} {token} wheel among {[w.name for w in self.wheels]}; pass "
"--all-wheel-pythons to verify every ABI in its own environment, or run from "
"a matching interpreter"
)
return [(tag, match, None)]
def _wheel_runtime(self) -> Step:
if self.options.no_runtime:
return Step(
"wheel-runtime",
True,
"WARNING: runtime verification SKIPPED via --no-runtime",
data={"status": "skipped"},
)
video = self.options.video or default_verification_video(self.repo_root)
if video is None:
return Step(
"wheel-runtime",
False,
"no verification clip found. Pass --video, set ORISYS_VERIFY_VIDEO, or commit a "
"clip under data/sample/. The runtime check is never silently skipped.",
)
video = Path(video)
if not video.is_file():
return Step("wheel-runtime", False, f"verification clip not found: {video}")
try:
targets = self._runtime_targets()
except RuntimeError as exc:
return Step("wheel-runtime", False, str(exc))
problems: list[str] = []
lines: list[str] = [f"clip: {video}", f"config: {self.options.config_name}"]
for tag, wheel, executable in targets:
label = f"{tag} {wheel.name}"
try:
digest = _sha256_file(wheel)
# One venv per wheel: it runs the import smoke test, the bundled
# asset check and the frame pipeline together.
verifier = verify_wheel
extra = {"python_executable": executable, "offline": self.options.offline}
if wheel_target(wheel)[1] != "win_amd64":
verifier = verify_manylinux_wheel
extra = {"repo_root": self.repo_root}
record = verifier(
wheel,
video=video,
config_name=self.options.config_name,
frames=self.options.frames,
check_assets=not self.options.no_assets,
**extra,
)
if _sha256_file(wheel) != digest:
raise RuntimeError(f"{wheel.name} changed during runtime verification")
except subprocess.CalledProcessError as exc:
problems.append(f"{label}: command failed ({exc})")
continue
except Exception as exc:
problems.append(f"{label}: {type(exc).__name__}: {exc}")
continue
lines.append(
f"{label}: {record['assets']} | frames={record['frames']} "
f"flow={record['flow_shape']}"
)
self.runtime_records.append({"tag": tag, **record, "sha256": digest})
detail = "\n".join(lines)
if problems:
detail += "\nfailures:\n" + "\n".join(problems)
return Step("wheel-runtime", not problems, detail, data={"records": self.runtime_records})
# -- driver ----------------------------------------------------------
def run(self) -> int:
started_at = _now()
started = time.perf_counter()
print(f"=== orisys release preflight: version {self.version} ({self.version_source}) ===")
print(f"repo: {self.repo_root}")
print(f"report: {self.report_dir}")
print()
try:
for step_fn in (
self._repo_state,
self._version_consistency,
self._version_strings,
self._duplicate_guard,
self._tests,
self._bundle_precheck,
self._wheel_inspection,
self._wheel_runtime,
):
self._execute(step_fn)
if self.aborted:
break
finally:
self.ok = bool(self.steps) and all(step.ok for step in self.steps) and not self.aborted
duration = round(time.perf_counter() - started, 3)
payload = {
"version": self.version,
"version_source": self.version_source,
"ok": self.ok,
"aborted": self.aborted,
"started_at": started_at,
"finished_at": _now(),
"duration_s": duration,
"git": self.git,
"interpreter": {
"executable": sys.executable,
"version": platform.python_version(),
"platform": sys.platform,
},
"options": {key: value for key, value in asdict(self.options).items()
if not isinstance(value, Path)},
"steps": [asdict(step) for step in self.steps],
"wheels": [
{
"name": wheel.name,
"size": wheel.stat().st_size,
"sha256": _sha256_file(wheel),
}
for wheel in self.wheels
],
}
payload["options"]["wheel_dir"] = str(self.options.wheel_dir) if self.options.wheel_dir else None
payload["options"]["video"] = str(self.options.video) if self.options.video else None
json_path, text_path = report_mod.write_report(payload, self.report_dir)
print()
print(f"report: {json_path}")
print(f"report: {text_path}")
print()
print(f"=== preflight {'PASSED' if self.ok else 'FAILED'} ({duration}s) ===")
return 0 if self.ok else 1
def run_preflight(options: PreflightOptions, repo_root: Path | None = None) -> int:
return Preflight(options, repo_root=repo_root).run()
+169
View File
@@ -0,0 +1,169 @@
"""Shared publish gate: certify a release from its preflight report.
Both publish commands (``release github`` and ``release pypi``) refuse to act
unless a passing ``release preflight`` report exists for the exact version and
every wheel still hashes to what that report recorded. The report is the contract
between "we verified this" and "we shipped this"; re-deriving the checks here
would just be a second, weaker opinion.
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass, field
from pathlib import Path
# Options that weaken the gate. A release must not be certified by a run that
# waived any of them: re-run preflight on a clean commit instead.
WAIVED_GUARD_KEYS = ("skip_tests", "allow_dirty", "force", "no_runtime", "no_assets")
class PublishGateError(RuntimeError):
"""The preflight report does not certify these wheels."""
@dataclass
class WheelRecord:
name: str
sha256: str
size: int = 0
@dataclass
class GateResult:
version: str
git_commit: str | None
report_path: Path
wheels: list[WheelRecord] = field(default_factory=list)
warnings: list[str] = field(default_factory=list)
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with Path(path).open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def evaluate_gate(
payload: dict,
version: str,
wheels: dict[str, Path],
*,
head_commit: str | None = None,
) -> tuple[list[str], list[str]]:
"""Return ``(problems, warnings)`` for a report payload and the wheels on disk.
Pure function: no I/O beyond hashing the wheels, so it is cheap to test.
"""
problems: list[str] = []
warnings: list[str] = []
if str(payload.get("version")) != version:
problems.append(
f"report is for version {payload.get('version')!r}, not {version!r}"
)
if not payload.get("ok"):
problems.append(
"report result is FAIL - run `release preflight --keep-going` and fix "
"everything before publishing"
)
if payload.get("aborted"):
problems.append("report stopped early (aborted); it certifies nothing")
options = payload.get("options") or {}
waived = sorted(key for key in WAIVED_GUARD_KEYS if options.get(key))
if waived:
problems.append(
"report waived gate options "
+ ", ".join(f"--{key.replace('_', '-')}" for key in waived)
+ "; commit the tree and re-run `release preflight` without them"
)
recorded = {item.get("name"): item.get("sha256") for item in payload.get("wheels") or []}
if not recorded:
problems.append("report lists no wheels")
runtime_records = {}
for step in payload.get("steps") or []:
if step.get("name") == "wheel-runtime" and step.get("ok"):
runtime_records.update({item.get("wheel"): item.get("sha256")
for item in (step.get("data") or {}).get("records", [])})
uncovered = sorted(name for name, digest in recorded.items()
if runtime_records.get(name) != digest)
if uncovered:
problems.append(f"wheels lack matching runtime verification: {uncovered}; "
"re-run preflight --all-wheel-pythons")
missing = sorted(set(recorded) - set(wheels))
if missing:
problems.append(f"wheels in the report are missing from disk: {missing}")
extra = sorted(set(wheels) - set(recorded))
if extra:
problems.append(f"wheels on disk are not covered by the report: {extra}")
for name in sorted(set(recorded) & set(wheels)):
actual = sha256_file(wheels[name])
if actual != recorded[name]:
problems.append(
f"{name} changed after preflight: report {recorded[name][:12]}, "
f"disk {actual[:12]} - rebuild and re-run preflight"
)
commit = payload.get("git", {}).get("commit") if isinstance(payload.get("git"), dict) else None
if head_commit and commit and commit != head_commit:
warnings.append(
f"report certifies commit {commit[:12]} but HEAD is {head_commit[:12]}; "
"the wheels are unchanged, so this is usually just a later commit"
)
return problems, warnings
def load_gate(repo_root: Path, version: str) -> GateResult:
"""Load and enforce the preflight report for ``version``.
Raises :class:`PublishGateError` with every problem found, so one run reports
all of them instead of stopping at the first.
"""
repo_root = Path(repo_root)
report_path = repo_root / "build" / "release" / "reports" / version / "preflight.json"
if not report_path.is_file():
raise PublishGateError(
f"no preflight report at {report_path}. Run:\n"
f" python build/run.py release preflight\n"
f"(the publish commands only ship what preflight certified)"
)
try:
payload = json.loads(report_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise PublishGateError(f"cannot read {report_path}: {exc}") from exc
wheel_dir = repo_root / "SDK" / version
wheels = {path.name: path for path in sorted(wheel_dir.glob("*.whl"))}
if not wheels:
raise PublishGateError(
f"no wheels in {wheel_dir}; build first (`python build/run.py wheel-win` / "
"`manylinux`) and re-run preflight"
)
problems, warnings = evaluate_gate(payload, version, wheels)
if problems:
raise PublishGateError(
f"the preflight report at {report_path} does not certify orisys {version}:\n "
+ "\n ".join(problems)
)
git_payload = payload.get("git") if isinstance(payload.get("git"), dict) else {}
return GateResult(
version=version,
git_commit=git_payload.get("commit"),
report_path=report_path,
wheels=[
WheelRecord(name=name, sha256=sha256_file(path), size=path.stat().st_size)
for name, path in wheels.items()
],
warnings=warnings,
)
+472
View File
@@ -0,0 +1,472 @@
"""Upload the certified wheels to TestPyPI, then (on confirmation) to PyPI.
``python build/run.py release pypi`` runs:
1. certified-wheels gate (same contract as ``release github``)
2. ``twine check`` on every wheel - the wheel metadata is generated, so its
long description and classifiers are worth validating before an upload that
cannot be undone
3. refuse if the version already exists on the target index (read-only probe)
4. upload to TestPyPI, then verify by fetching it back from the index
5. stop there unless ``--confirm-pypi`` is given; then repeat 3-4 against PyPI
PyPI uploads are permanent: a version number can never be reused, and a bad
artifact cannot be replaced. Hence the TestPyPI stage, the read-only existence
probe, and ``--apply`` being required even to reach TestPyPI.
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import time
import urllib.error
import urllib.request
from dataclasses import dataclass, field
from pathlib import Path
_BUILD_ROOT = Path(__file__).resolve().parents[1]
if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import REPO_ROOT # noqa: E402
from release import auth # noqa: E402
from release.publish import GateResult, PublishGateError, load_gate, sha256_file # noqa: E402
INDEX_JSON = {name: str(meta["json_url"]) for name, meta in auth.INDEXES.items()}
INDEX_SIMPLE = {name: str(meta["simple_url"]) for name, meta in auth.INDEXES.items()}
@dataclass
class Options:
version: str
apply: bool = False
confirm_pypi: bool = False
verify_install: bool = True
skip_check: bool = False
# Upload straight to PyPI, skipping the TestPyPI rehearsal. Needed when you
# only hold a production token, since the two indexes reject each other's.
pypi_only: bool = False
# Offer to enter a missing token now instead of just failing.
prompt_credentials: bool = False
# Token supplied non-interactively (e.g. --token-stdin), used once.
token: str | None = None
def upload_phases(options: Options) -> list[str]:
"""The indexes to upload to, in order."""
if options.pypi_only:
return ["pypi"]
return ["testpypi", "pypi"] if options.confirm_pypi else ["testpypi"]
@dataclass
class IndexState:
versions: list[str] = field(default_factory=list)
error: str | None = None
@property
def known(self) -> bool:
return self.error is None
def twine_check_argv(wheels: list[Path]) -> list[str]:
return [sys.executable, "-m", "twine", "check", *[str(w) for w in wheels]]
def twine_upload_argv(wheels: list[Path], repository: str | None) -> list[str]:
argv = [sys.executable, "-m", "twine", "upload"]
if repository:
argv += ["--repository", repository]
argv += [str(wheel) for wheel in wheels]
return argv
def credentials_hint(index: str) -> str | None:
"""Guidance when ``index`` has no usable credentials, else ``None``.
More than "does the file exist": a placeholder or the wrong ``username`` is
treated as missing, because twine would only find out at upload time.
"""
state = auth.credential_state(index)
if state.configured:
return None
lines = [f"no usable credentials for [{index}]: {state.problem}"]
lines.append(f" set it up with: python build/run.py release auth --set-token {index}")
if index == "testpypi":
lines.append(
" (TestPyPI needs its own account and token; if you only hold a "
"production token, use --pypi-only to skip the rehearsal)"
)
lines.append(
" or set TWINE_USERNAME=__token__ and TWINE_PASSWORD for this session only"
)
return "\n".join(lines)
def _ensure_credentials(
indexes: list[str], options: Options, *, interactive: bool | None = None
) -> bool:
"""Make sure every index that will be uploaded to has credentials.
Prompts for a missing token when there is a terminal to prompt on; a piped or
CI run fails fast instead of hanging on a hidden-input prompt.
"""
missing = [index for index in indexes if not auth.credential_state(index).configured]
if not missing:
for index in indexes:
state = auth.credential_state(index)
print(f"[ok ] credentials for [{index}] ({state.source})")
return True
if interactive is None:
interactive = sys.stdin.isatty()
allow_prompt = options.prompt_credentials or (interactive and options.apply)
if not allow_prompt:
for index in missing:
print(f"[{'FAIL' if options.apply else 'warn'}] {credentials_hint(index)}")
if not options.apply:
print(
"[info] dry run continues so you can review the plan; "
"--apply would stop here until the credentials exist"
)
return True
if not interactive:
print(
"[info] stdin is not a terminal, so nothing was prompted; run "
"`release auth --set-token <index>` or pass --prompt-credentials"
)
return False
for index in missing:
print(f"[warn] {credentials_hint(index)}")
token = options.token or auth.prompt_for_token(index)
if not token:
print(f"[FAIL] no token entered for [{index}]")
return False
for problem in auth.validate_token(token):
print(f"[FAIL] {problem.removeprefix('ERROR ')}")
return False
for warning in auth.token_warnings(token):
print(f"[warn] {warning.removeprefix('WARN ')}")
path = auth.write_token(index, token)
print(f"[ok ] wrote [{index}] credentials to {path}")
options.token = None # only the first index may consume a piped token
return True
def index_state(index: str, *, timeout: float = 20.0) -> IndexState:
"""Read the published versions of orisys from an index (read-only)."""
url = INDEX_JSON[index]
request = urllib.request.Request(
url, headers={"User-Agent": "orisys-release-publish", "Accept": "application/json"}
)
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
payload = json.load(response)
except urllib.error.HTTPError as exc:
if exc.code == 404:
return IndexState(versions=[])
return IndexState(error=f"{index} returned HTTP {exc.code}")
except Exception as exc: # network, DNS, TLS, JSON
return IndexState(error=f"{index} lookup failed: {type(exc).__name__}: {exc}")
releases = payload.get("releases")
if not isinstance(releases, dict):
return IndexState(error=f"unexpected {index} response (no 'releases' mapping)")
return IndexState(
versions=sorted(name for name, files in releases.items() if files)
)
def index_files(
index: str, version: str, *, timeout: float = 20.0
) -> tuple[list[str] | None, str | None]:
"""Filenames published for ``version``, as ``(filenames, error)``.
``(None, None)`` - the index does not serve this version (yet)
``(None, "why")`` - the lookup itself failed, which is NOT evidence of absence
``([...], None)`` - the version is published with those files
Keeping those apart matters: reporting a network hiccup as "not published
yet" is how a successful upload gets mistaken for a failure.
"""
url = INDEX_JSON[index]
request = urllib.request.Request(
url, headers={"User-Agent": "orisys-release-publish", "Accept": "application/json"}
)
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
payload = json.load(response)
except urllib.error.HTTPError as exc:
if exc.code == 404:
return None, None
return None, f"{index} returned HTTP {exc.code}"
except Exception as exc:
return None, f"{index} lookup failed: {type(exc).__name__}: {exc}"
files = (payload.get("releases") or {}).get(version)
if not isinstance(files, list) or not files:
return None, None
return sorted(str(item.get("filename", "")) for item in files), None
def wait_for_release(
index: str,
version: str,
*,
attempts: int = 6,
delay: float = 5.0,
sleep=None,
timeout: float = 20.0,
) -> tuple[list[str] | None, str | None]:
"""Poll until the index serves ``version`` (uploads are not instantly visible).
TestPyPI in particular lags the upload response by seconds. Returns the last
``(filenames, error)`` observed.
"""
sleep = sleep or time.sleep
last_error: str | None = None
for attempt in range(1, max(1, attempts) + 1):
filenames, error = index_files(index, version, timeout=timeout)
if filenames is not None:
return filenames, None
last_error = error
if attempt < attempts:
print(
f"[info] {index} does not serve {version} yet "
f"({'lookup failed' if error else 'indexing'}), retry {attempt}/{attempts - 1}",
flush=True,
)
sleep(delay)
return None, last_error
def existing_release_state(index: str, version: str, expected: set[str]) -> str:
"""``"satisfied"`` | ``"conflict"`` | ``"absent"`` for this index and version.
An index version can never be replaced, so "already published with exactly
these files" is the only successful terminal state; treating it as an error
would make a re-run after a partial failure look like a new failure. A
different file set is a genuine conflict and must stop the run.
"""
published, error = index_files(index, version)
if published is None:
if error:
print(f"[warn] could not check {index} for an existing release: {error}")
return "absent"
if set(published) == expected:
print(
f"[ok ] {index} already serves orisys {version} with all "
f"{len(published)} file(s)"
)
return "satisfied"
extra = sorted(set(published) - expected)
missing = sorted(expected - set(published))
print(
f"[FAIL] {index} already serves orisys {version} with a different file set"
+ (f"; unexpected: {extra}" if extra else "")
+ (f"; missing: {missing}" if missing else "")
)
return "conflict"
def _upload(wheels: list[Path], repository: str | None) -> int:
argv = twine_upload_argv(wheels, repository)
print(f"[run ] {' '.join(str(part) for part in argv)}")
completed = subprocess.run(argv, check=False)
return completed.returncode
def _fetch_back(index: str, version: str, wheel: Path) -> int:
"""Fetch the exact Python/platform wheel and verify its certified bytes."""
with tempfile.TemporaryDirectory(prefix="orisys-fetch-") as tmp:
target = Path(tmp) / "out"
python_tag, abi_tag, platform_tags = wheel.stem.rsplit("-", 3)[1:]
argv = [
sys.executable,
"-m",
"pip",
"download",
"--no-deps",
"--no-cache-dir",
"--only-binary=:all:",
"--implementation", "cp",
"--python-version", python_tag[2:],
"--abi", abi_tag,
"--index-url",
INDEX_SIMPLE[index],
"--dest",
str(target),
f"orisys=={version}",
]
for platform_tag in platform_tags.split("."):
argv += ["--platform", platform_tag]
# Index verification must not inherit a fallback index or wheelhouse.
env = {key: value for key, value in os.environ.items() if not key.startswith("PIP_")}
env["PIP_CONFIG_FILE"] = os.devnull
print(f"[run ] {' '.join(argv)}")
completed = subprocess.run(argv, check=False, env=env)
if completed.returncode != 0:
return completed.returncode
fetched = sorted(target.glob(f"orisys-{version}*"))
if not fetched:
print(f"[FAIL] pip download produced nothing for orisys=={version}")
return 1
print(f"[ok ] fetched {[p.name for p in fetched]} from {index}")
if {path.name for path in fetched} != {wheel.name}:
raise PublishGateError(f"{index} returned a different wheel than {wheel.name}")
if sha256_file(fetched[0]) != sha256_file(wheel):
raise PublishGateError(f"{index} returned different bytes for {wheel.name}")
return 0
def publish(options: Options) -> int:
version = options.version
mode = "APPLY" if options.apply else "DRY RUN"
print(f"=== release pypi: orisys {version} ({mode}) ===")
# --- 1. gate -------------------------------------------------------
try:
gate: GateResult = load_gate(REPO_ROOT, version)
except PublishGateError as exc:
print(f"[FAIL] {exc}")
return 1
print(f"[ok ] preflight report: {gate.report_path}")
for wheel in gate.wheels:
print(f" [ok] {wheel.name} sha256={wheel.sha256[:12]}")
for warning in gate.warnings:
print(f"[warn] {warning}")
wheels = [REPO_ROOT / "SDK" / version / record.name for record in gate.wheels]
missing = [str(wheel) for wheel in wheels if not wheel.is_file()]
if missing:
print(f"[FAIL] wheels missing from disk: {missing}")
return 1
# --- 2. credentials + twine check (before anything irreversible) ----
phases = upload_phases(options)
if options.pypi_only:
print(
"[warn] --pypi-only: uploading straight to PyPI with no TestPyPI "
"rehearsal. An index version can never be reused or replaced."
)
if not _ensure_credentials(phases, options):
return 1
if not options.skip_check:
argv = twine_check_argv(wheels)
print(f"[run ] {' '.join(str(part) for part in argv)}")
completed = subprocess.run(argv, check=False)
if completed.returncode != 0:
print("[FAIL] twine check reported problems; fix the wheel metadata first")
return 1
print("[ok ] twine check passed")
# --- 3. phase plan -------------------------------------------------
expected = {wheel.name for wheel in wheels}
remaining: list[str] = []
for index in phases:
state = existing_release_state(index, version, expected)
if state == "conflict":
return 1
if state == "absent":
remaining.append(index)
phases = remaining
print()
for index in phases:
state = index_state(index)
if not state.known:
print(f"[FAIL] {state.error}; cannot safely check for duplicate uploads")
return 1
if version in state.versions:
print(
f"[FAIL] orisys {version} already exists on {index} "
f"(published: {', '.join(state.versions) or 'none'}) - "
"an index version cannot be reused"
)
return 1
print(f"[ok ] {index}: {version} not present (published: {', '.join(state.versions) or 'none'})")
if not phases:
print()
print("=== release pypi: every requested index already has this release ===")
return 0
print()
for index in phases:
print(f"[plan] upload {len(wheels)} wheel(s) to {index}")
if phases == ["testpypi"]:
print(
"[plan] stop after TestPyPI. Re-run with "
"`--confirm-pypi --apply` to publish to the real PyPI."
)
print()
if not options.apply:
print("dry run: nothing was uploaded. Re-run with --apply to execute.")
return 0
# --- 4. upload ----------------------------------------------------
for index in phases:
if _upload(wheels, index) != 0:
print(f"[FAIL] twine upload to {index} failed")
return 1
print(f"[ok ] uploaded to {index}")
published, error = wait_for_release(index, version)
if published is None:
# The upload itself succeeded, so this is a confirmation problem, not
# a publishing failure. Say so plainly and do not imply data loss.
reason = f"lookup kept failing ({error})" if error else "the index has not indexed it yet"
print(
f"[warn] uploaded, but could not confirm it on {index} yet: {reason}.\n"
f" check https://{'test.' if index == 'testpypi' else ''}pypi.org/project/orisys/{version}/"
"\n re-running this command is safe: an already-published version is skipped"
)
print(f"\n=== release pypi: {index} upload SUCCEEDED (verification inconclusive) ===")
return 0
if set(published) != expected:
print(
f"[FAIL] {index} file set mismatch:\n expected {sorted(expected)}\n"
f" got {sorted(published)}"
)
return 1
print(f"[ok ] {index} serves all {len(published)} file(s) for {version}")
if options.verify_install:
fetch_pending = False
for wheel in wheels:
try:
if _fetch_back(index, version, wheel) != 0:
fetch_pending = True
print(f"[warn] {wheel.name} is uploaded but download verification is inconclusive")
else:
print(f"[ok ] fetched and hash-verified {wheel.name} from {index}")
except PublishGateError as exc:
print(f"[FAIL] {exc}")
return 1
if fetch_pending:
print(f"[warn] {index} upload succeeded but download verification is inconclusive; "
"later indexes were not uploaded. Re-run after checking propagation.")
return 0
if phases == ["testpypi"]:
print()
print("=== release pypi: TestPyPI stage PASSED; PyPI untouched ===")
print("next: python build/run.py release pypi --confirm-pypi --apply")
return 0
print()
if options.pypi_only:
print("=== release pypi PASSED (PyPI only, no TestPyPI rehearsal) ===")
else:
print("=== release pypi PASSED (TestPyPI + PyPI) ===")
return 0
+71
View File
@@ -0,0 +1,71 @@
"""Structured preflight reports.
A preflight run writes both a machine-readable ``preflight.json`` (the contract a
future publish step will consume and re-verify) and a human ``preflight.txt``.
"""
from __future__ import annotations
import json
from pathlib import Path
REPORTS_RELPATH = Path("build") / "release" / "reports"
SCHEMA_VERSION = 1
def reports_root(repo_root: Path, override: Path | None = None) -> Path:
return Path(override) if override else Path(repo_root) / REPORTS_RELPATH
def render_text(payload: dict) -> str:
git = payload.get("git") or {}
interpreter = payload.get("interpreter") or {}
lines = [
f"orisys release preflight :: version {payload.get('version')}",
f" result: {'PASS' if payload.get('ok') else 'FAIL'}",
f" git: {git.get('branch')} @ {git.get('commit')} (dirty={git.get('dirty')})",
f" interpreter: {interpreter.get('version')} at {interpreter.get('executable')}",
f" duration: {payload.get('duration_s')}s",
"",
]
for step in payload.get("steps", []):
flag = "ok " if step.get("ok") else "FAIL"
lines.append(f"[{flag}] {step.get('name')} ({step.get('duration_s')}s)")
for detail_line in str(step.get("detail") or "").splitlines():
lines.append(f" {detail_line}")
lines.append("")
wheels = payload.get("wheels") or []
if wheels:
lines.append("wheels:")
for wheel in wheels:
lines.append(
f" {wheel.get('name')} {wheel.get('size')} bytes sha256={wheel.get('sha256')}"
)
lines.append("")
options = payload.get("options") or {}
waived = [
key
for key, value in options.items()
if value and key in {"skip_tests", "allow_dirty", "force", "no_runtime", "no_assets"}
]
if waived:
lines.append(f"waived guards: {', '.join(sorted(waived))}")
lines.append("")
return "\n".join(lines).rstrip() + "\n"
def write_report(payload: dict, out_dir: Path) -> tuple[Path, Path]:
out_dir = Path(out_dir)
out_dir.mkdir(parents=True, exist_ok=True)
payload = {**payload, "schema": SCHEMA_VERSION}
json_path = out_dir / "preflight.json"
text_path = out_dir / "preflight.txt"
json_path.write_text(json.dumps(payload, indent=2, sort_keys=False), encoding="utf-8")
text_path.write_text(render_text(payload), encoding="utf-8")
return json_path, text_path
+117
View File
@@ -0,0 +1,117 @@
"""Run Linux wheel release checks in the matching manylinux Docker image."""
from __future__ import annotations
import json
import subprocess
from pathlib import Path
from packaging.utils import parse_wheel_filename
PLATFORM_LABELS = {
"win_amd64": "Windows x64",
"x86_64": "Linux x86_64",
"aarch64": "Linux ARM64",
}
def wheel_target(wheel: Path) -> tuple[str, str]:
"""Return the CPython tag and supported OS/architecture; reject ambiguity."""
_, _, _, tags = parse_wheel_filename(wheel.name)
identities = set()
for tag in tags:
if (
tag.interpreter not in {"cp310", "cp311", "cp312"}
or tag.abi != tag.interpreter
):
raise RuntimeError(f"Unsupported Python ABI in {wheel.name}")
if tag.platform == "win_amd64":
target = "win_amd64"
elif tag.platform.startswith("manylinux") and tag.platform.endswith("_aarch64"):
target = "aarch64"
elif tag.platform.startswith("manylinux") and tag.platform.endswith("_x86_64"):
target = "x86_64"
else:
raise RuntimeError(f"Unsupported release platform in {wheel.name}")
identities.add((tag.interpreter, target))
if len(identities) != 1:
raise RuntimeError(f"Ambiguous release platform in {wheel.name}")
return identities.pop()
_CONTAINER_CHECK = """
import json, sys
from pathlib import Path
sys.path.insert(0, '/src/build')
from common.verify.wheel import verify_wheel
record = verify_wheel(Path(sys.argv[1]), video=Path(sys.argv[2]),
config_name=sys.argv[3], frames=int(sys.argv[4]), check_assets=sys.argv[5] == '1')
print('ORISYS_RUNTIME_RESULT=' + json.dumps(record))
"""
def verify_manylinux_wheel(
wheel: Path,
*,
repo_root: Path,
video: Path,
config_name: str,
frames: int,
check_assets: bool = True,
) -> dict:
tag, architecture = wheel_target(wheel)
if architecture == "win_amd64":
raise RuntimeError("Windows wheels must be verified with a Windows interpreter")
platform = "linux/arm64" if architecture == "aarch64" else "linux/amd64"
image = (
"orisys-manylinux-aarch64" if architecture == "aarch64" else "orisys-manylinux"
)
abi = f"{tag}-{tag}"
argv = [
"docker",
"run",
"--rm",
"--pull=never",
"--platform",
platform,
"-v",
f"{repo_root.resolve()}:/src:ro",
"-v",
f"{wheel.resolve().parent}:/wheels:ro",
"-v",
f"{video.resolve().parent}:/video:ro",
"-e",
"PIP_NO_INDEX=1",
"-e",
f"PIP_FIND_LINKS=/opt/orisys-wheelhouse/{abi}",
"--entrypoint",
f"/opt/python/{abi}/bin/python",
image,
"-c",
_CONTAINER_CHECK,
f"/wheels/{wheel.name}",
f"/video/{video.name}",
config_name,
str(frames),
"1" if check_assets else "0",
]
print(f" verifying {wheel.name} in {image} ({platform}) ...", flush=True)
result = subprocess.run(argv, capture_output=True, text=True, check=False)
if result.returncode:
raise RuntimeError(
f"{image} verification failed:\n{result.stdout}\n{result.stderr}"
)
marker = "ORISYS_RUNTIME_RESULT="
records = [
line[len(marker) :]
for line in result.stdout.splitlines()
if line.startswith(marker)
]
if len(records) != 1:
raise RuntimeError(
f"{image} returned no unique verification record:\n{result.stdout}"
)
record = json.loads(records[0])
if record.get("wheel") != wheel.name or record.get("frames") != frames:
raise RuntimeError(f"Unexpected verification record for {wheel.name}: {record}")
return record
+272
View File
@@ -0,0 +1,272 @@
"""Version consistency and duplicate-release guards for the release gate."""
from __future__ import annotations
import json
import os
import re
import subprocess
import urllib.error
import urllib.request
from pathlib import Path
PYPI_JSON_URL = "https://pypi.org/pypi/orisys/json"
# Sibling repository that carries the public GitHub releases and their tags.
RELEASE_REPO_DIRNAME = "orisys-sdk-release"
# orisys-<version>-<pytag>-<abitag>-<platformtag>.whl
_WHEEL_NAME_RE = re.compile(r"^orisys-(?P<version>[^-]+)-[^-]+-[^-]+-.+\.whl$")
def wheel_version(wheel_path: Path) -> str | None:
match = _WHEEL_NAME_RE.match(Path(wheel_path).name)
return match.group("version") if match else None
def sdk_versions(repo_root: Path) -> list[str]:
"""Version directory names present under ``SDK/``, in ascending order.
Only digit-leading names count as versions, so stray directories (``tmp``,
``latest``) never appear as releases.
"""
sdk_root = Path(repo_root) / "SDK"
if not sdk_root.is_dir():
return []
return sorted(
(
entry.name
for entry in sdk_root.iterdir()
if entry.is_dir() and entry.name[:1].isdigit()
),
key=_version_sort_key,
)
def sdk_wheels(repo_root: Path, version: str) -> list[Path]:
"""Wheels already published into ``SDK/<version>/``."""
version_dir = Path(repo_root) / "SDK" / version
if not version_dir.is_dir():
return []
return sorted(version_dir.glob("*.whl"))
def dynamic_version_attr(repo_root: Path) -> str | None:
"""Return pyproject's configured dynamic version attr (e.g. orisys.__version__)."""
from common.pyproject_utils import load_pyproject
config = (
load_pyproject(Path(repo_root))
.get("tool", {})
.get("setuptools", {})
.get("dynamic", {})
.get("version")
)
if isinstance(config, dict):
attr = config.get("attr")
if isinstance(attr, str):
return attr
return None
def pypi_released_versions(timeout: float = 20.0) -> tuple[list[str] | None, str | None]:
"""Read the published version list from PyPI.
Returns ``(versions, error)``; ``versions`` is ``None`` when the lookup
failed. This is the only network call in the release gate and it is
read-only, so it is safe to run before an irreversible upload.
"""
request = urllib.request.Request(
PYPI_JSON_URL,
headers={"User-Agent": "orisys-release-preflight", "Accept": "application/json"},
)
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
payload = json.load(response)
except urllib.error.HTTPError as exc:
if exc.code == 404:
return [], None
return None, f"PyPI returned HTTP {exc.code}"
except Exception as exc: # network, DNS, TLS, JSON
return None, f"{type(exc).__name__}: {exc}"
releases = payload.get("releases")
if not isinstance(releases, dict):
return None, "unexpected PyPI response shape (no 'releases' mapping)"
listed = [
str(version)
for version, files in releases.items()
if isinstance(files, list) and files
]
return sorted(listed, key=_version_sort_key), None
def _version_sort_key(name: str):
parts = re.findall(r"\d+", name)
return ([int(part) for part in parts] or [0], name)
def git_tags(repo_root: Path) -> list[str]:
"""Local git tags from this repo and the sibling release repo.
Tags are the only offline evidence that a version was already released
(``v0.5.1`` in ``orisys-sdk-release`` marks a published GitHub release), so
the duplicate guard works without network access.
"""
roots = [Path(repo_root)]
sibling = Path(repo_root).parent / RELEASE_REPO_DIRNAME
if sibling.is_dir():
roots.append(sibling)
tags: set[str] = set()
for root in roots:
try:
completed = subprocess.run(
["git", "-C", str(root), "tag", "--list"],
capture_output=True,
text=True,
check=False,
)
except OSError:
continue
if completed.returncode != 0:
continue
tags.update(
line.strip() for line in (completed.stdout or "").splitlines() if line.strip()
)
return sorted(tags)
def released_tags_for(version: str, tags: list[str]) -> list[str]:
"""Tag spellings that mark ``version`` as already released."""
return sorted({version, f"v{version}"} & set(tags))
# Version literals that are deliberately not the package version. Keyed by
# repo-relative path; each entry documents *why* the literal may stay.
VERSION_STRING_ALLOWLIST: dict[str, frozenset[str]] = {
# Historical release records: rewriting these would destroy the audit trail.
"docs/publishing.md": frozenset({"0.4.2", "0.4.3", "0.4.4", "0.5.0", "0.5.1", "0.5.2"}),
# Unrelated third-party/CUDA versions quoted in prose.
"docs/get_started.md": frozenset({"12.9.0"}),
}
# Paths never scanned: tests deliberately use synthetic versions, reports are
# generated output, and this module's own allowlist/sentinel literals would
# otherwise flag themselves.
VERSION_SCAN_EXCLUDED_PREFIXES = (
"tests/",
"build/release/reports/",
"build/nuitka/",
"build/lib/",
"dist/",
"SDK/",
"data/",
)
VERSION_SCAN_EXCLUDED_FILES = ("build/release/version.py",)
_VERSION_LITERAL_RE = re.compile(
r"(?<![<>=!~\w.])v?(\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.]+)?)(?![0-9A-Za-z.])"
)
_SCANNED_SUFFIXES = (".md", ".py", ".txt", ".toml", ".cfg", ".ini", ".rst")
def tracked_text_files(repo_root: Path) -> list[Path]:
"""Tracked files worth scanning for version literals."""
repo_root = Path(repo_root)
try:
completed = subprocess.run(
["git", "-C", str(repo_root), "ls-files", "-z"],
capture_output=True,
text=True,
check=False,
)
names = [n for n in (completed.stdout or "").split("\0") if n]
except OSError:
names = []
if not names:
names = [
str(path.relative_to(repo_root)).replace("\\", "/")
for path in repo_root.rglob("*")
if path.is_file()
]
selected = []
for name in names:
if name.startswith(VERSION_SCAN_EXCLUDED_PREFIXES):
continue
if name in VERSION_SCAN_EXCLUDED_FILES:
continue
if not name.endswith(_SCANNED_SUFFIXES):
continue
selected.append(Path(name))
return selected
def find_version_literals(
repo_root: Path, version: str
) -> list[tuple[str, int, str]]:
"""``(path, lineno, literal)`` for every version-looking string != ``version``.
Dependency constraints are skipped by the regex lookbehind (``numpy>=1.24.4``
is not a version reference), and historical/third-party literals require an
explicit entry in :data:`VERSION_STRING_ALLOWLIST`.
"""
repo_root = Path(repo_root)
findings: list[tuple[str, int, str]] = []
for relative in tracked_text_files(repo_root):
absolute = repo_root / relative
try:
text = absolute.read_text(encoding="utf-8", errors="ignore")
except OSError:
continue
if "\0" in text:
continue
allowed = VERSION_STRING_ALLOWLIST.get(relative.as_posix(), frozenset())
for lineno, line in enumerate(text.splitlines(), start=1):
for match in _VERSION_LITERAL_RE.finditer(line):
literal = match.group(1)
if literal == version or literal in allowed:
continue
findings.append((relative.as_posix(), lineno, literal))
return findings
def conda_envs_root() -> Path | None:
"""Locate the conda envs directory the same way build/wheel/win.bat does."""
override = os.getenv("CONDA_ENVS_ROOT")
if override:
path = Path(override)
return path if path.is_dir() else None
home = Path.home()
candidates = (
home / "anaconda3" / "envs",
home / "miniconda3" / "envs",
Path(os.getenv("LOCALAPPDATA", "")) / "anaconda3" / "envs",
Path(os.getenv("LOCALAPPDATA", "")) / "miniconda3" / "envs",
)
for candidate in candidates:
if candidate.is_dir():
return candidate
return None
def python_for_tag(tag: str, envs_root: Path | None = None) -> Path | None:
"""Resolve an interpreter for a wheel tag like ``cp310`` (env ``py310``)."""
digits = tag[2:] if tag.startswith("cp") else tag
env_name = f"py{digits}"
root = envs_root or conda_envs_root()
if root is None:
return None
for relative in (
Path(env_name) / "python.exe",
Path(env_name) / "bin" / "python",
):
candidate = root / relative
if candidate.is_file():
return candidate
return None
+17 -3
View File
@@ -8,6 +8,8 @@ Usage (from repo root):
python build/run.py gui finger|mini|box python build/run.py gui finger|mini|box
python build/run.py verify [wheel.whl] python build/run.py verify [wheel.whl]
python build/run.py verify --gui-dist [--product ory_finger] python build/run.py verify --gui-dist [--product ory_finger]
python build/run.py release preflight [--version V]
python build/run.py release audit <wheel.whl>
See build/README.md for details. See build/README.md for details.
""" """
@@ -74,15 +76,23 @@ def _cmd_verify(argv: list[str]) -> int:
return verify_main(argv) return verify_main(argv)
def _cmd_release(argv: list[str]) -> int:
from release.cli import main as release_main
return release_main(argv)
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
argv = list(sys.argv[1:] if argv is None else argv) argv = list(sys.argv[1:] if argv is None else argv)
if not argv or argv[0] in ("-h", "--help"): if not argv or argv[0] in ("-h", "--help"):
print(__doc__) print(__doc__)
print("Commands: wheel | wheel-win | manylinux | gui | verify") print("Commands: wheel | wheel-win | wheel-manylinux | gui | verify | release")
print("Examples:") print("Examples:")
print(" python build/run.py wheel --help") print(" python build/run.py wheel --help")
print(" python build/run.py gui finger") print(" python build/run.py gui finger")
print(" python build/run.py manylinux -PythonVersions 3.10") print(" python build/run.py wheel-manylinux -Architecture aarch64 -PythonVersions 3.10")
print(" python build/run.py release preflight --version <version>")
print(" python build/run.py release audit SDK\\<version>\\orisys-<version>-cp312-cp312-win_amd64.whl")
return 0 return 0
command, rest = argv[0], argv[1:] command, rest = argv[0], argv[1:]
@@ -92,10 +102,14 @@ def main(argv: list[str] | None = None) -> int:
"wheel-manylinux": _cmd_manylinux, "wheel-manylinux": _cmd_manylinux,
"gui": _cmd_gui, "gui": _cmd_gui,
"verify": _cmd_verify, "verify": _cmd_verify,
"release": _cmd_release,
} }
if command not in dispatch: if command not in dispatch:
print(f"Unknown command: {command!r}", file=sys.stderr) print(f"Unknown command: {command!r}", file=sys.stderr)
print("Choose from: wheel, wheel-win, manylinux, gui, verify", file=sys.stderr) print(
"Choose from: wheel, wheel-win, wheel-manylinux, gui, verify, release",
file=sys.stderr,
)
return 2 return 2
return dispatch[command](rest) return dispatch[command](rest)
+27 -11
View File
@@ -29,9 +29,22 @@ _BUILD_ROOT = Path(__file__).resolve().parents[1]
if str(_BUILD_ROOT) not in sys.path: if str(_BUILD_ROOT) not in sys.path:
sys.path.insert(0, str(_BUILD_ROOT)) sys.path.insert(0, str(_BUILD_ROOT))
from common.paths import NUITKA_COMPILE_DIR, NUITKA_WHEEL_DIR, REPO_ROOT, repaired_wheel_dir, repo_path from common.paths import (
from common.pyproject_utils import bundle_items, extras_require, get_version, install_requires NUITKA_COMPILE_DIR,
NUITKA_WHEEL_DIR,
REPO_ROOT,
repaired_wheel_dir,
repo_path,
)
from common.pyproject_utils import (
bundle_items,
extras_require,
get_version,
install_requires,
project_metadata,
)
from common.verify.wheel import verify_built_wheel from common.verify.wheel import verify_built_wheel
from wheel.compile import compile_orisys, copy_compiled_orisys, install_nuitka from wheel.compile import compile_orisys, copy_compiled_orisys, install_nuitka
from wheel.layout import ( from wheel.layout import (
copy_bundle_items, copy_bundle_items,
@@ -41,6 +54,7 @@ from wheel.layout import (
from wheel.package import ( from wheel.package import (
is_compliant_manylinux_wheel, is_compliant_manylinux_wheel,
publish_wheel_to_sdk, publish_wheel_to_sdk,
read_long_description,
repair_linux_wheel, repair_linux_wheel,
run_bdist_wheel, run_bdist_wheel,
write_wheel_setup, write_wheel_setup,
@@ -62,7 +76,7 @@ def _validate_pypi_linux_environment() -> None:
if sys.platform != "linux": if sys.platform != "linux":
raise SystemExit( raise SystemExit(
"[ERROR] --target pypi-linux must run inside a Linux manylinux container " "[ERROR] --target pypi-linux must run inside a Linux manylinux container "
"(for example quay.io/pypa/manylinux_2_28_x86_64 via Docker Desktop)." "(manylinux_2_28_x86_64 or manylinux_2_28_aarch64 via Docker Desktop)."
) )
if sys.version_info[:2] not in PYPI_LINUX_PYTHON_VERSIONS: if sys.version_info[:2] not in PYPI_LINUX_PYTHON_VERSIONS:
supported = ", ".join(f"{major}.{minor}" for major, minor in PYPI_LINUX_PYTHON_VERSIONS) supported = ", ".join(f"{major}.{minor}" for major, minor in PYPI_LINUX_PYTHON_VERSIONS)
@@ -111,10 +125,12 @@ def build(*, target: str = "internal", verify: bool | None = None) -> Path:
# Bundle rules live in pyproject.toml → [[tool.orisys.bundle.items]] # Bundle rules live in pyproject.toml → [[tool.orisys.bundle.items]]
bundle = bundle_items(REPO_ROOT) bundle = bundle_items(REPO_ROOT)
copied_counts = copy_bundle_items(REPO_ROOT, pkg_dir, bundle) copied_counts = copy_bundle_items(REPO_ROOT, pkg_dir, bundle)
print(f"Copied {copied_counts['configs']} JSON config files") for item in bundle:
print(f"Copied {copied_counts['examples']} example files to orisys/examples/") destination = f"orisys/{item['target']}/"
print(f"Copied {copied_counts['docs']} doc files to orisys/docs/") print(
print(f"Copied {copied_counts['sensors']} sensor bundle files to orisys/sensors/") f"Copied {copied_counts.get(item['name'], 0)} file(s) "
f"for bundle item {item['name']!r} to {destination}"
)
write_wheel_setup( write_wheel_setup(
NUITKA_WHEEL_DIR, NUITKA_WHEEL_DIR,
@@ -122,6 +138,8 @@ def build(*, target: str = "internal", verify: bool | None = None) -> Path:
package_data=package_data_from_bundle(ext_suffix, bundle), package_data=package_data_from_bundle(ext_suffix, bundle),
install_requires=install_requires(REPO_ROOT), install_requires=install_requires(REPO_ROOT),
extras_require=extras_require(REPO_ROOT), extras_require=extras_require(REPO_ROOT),
project=project_metadata(REPO_ROOT),
long_description=read_long_description(REPO_ROOT),
) )
# --- 3. Build wheel, repair on Linux, publish to SDK/ --- # --- 3. Build wheel, repair on Linux, publish to SDK/ ---
@@ -135,7 +153,7 @@ def build(*, target: str = "internal", verify: bool | None = None) -> Path:
if target == "pypi-linux" and not is_compliant_manylinux_wheel(final_candidate): if target == "pypi-linux" and not is_compliant_manylinux_wheel(final_candidate):
raise SystemExit( raise SystemExit(
f"[ERROR] Wheel is not PyPI-compliant: {final_candidate.name}. " f"[ERROR] Wheel is not PyPI-compliant: {final_candidate.name}. "
"Expected a manylinux_2_17_x86_64 (or newer) platform tag." "Expected a supported manylinux x86_64 or aarch64 platform tag."
) )
# Keep a durable copy under dist/repaired for upload tooling. # Keep a durable copy under dist/repaired for upload tooling.
@@ -188,9 +206,7 @@ def main(argv: list[str] | None = None) -> None:
) )
args = parser.parse_args(argv) args = parser.parse_args(argv)
verify: bool | None verify: bool | None
if args.target == "pypi-linux": if args.target == "pypi-linux" or args.verify:
verify = True
elif args.verify:
verify = True verify = True
elif args.skip_verify: elif args.skip_verify:
verify = False verify = False
+8
View File
@@ -32,12 +32,20 @@ NUITKA_NOFOLLOW_IMPORTS = (
"cv2", "cv2",
"numba", "numba",
"orisys_blend", "orisys_blend",
# Optional accelerators / GUIs: imported behind try/except, resolved from the
# user's environment at runtime. Listing them here is not optional - without
# it, Nuitka follows whatever happens to be installed in the *build*
# environment, which blows up compile time and makes wheel contents depend on
# ambient packages. tests/test_nuitka_nofollow.py enforces full coverage.
"cupy", "cupy",
"cupyx", "cupyx",
"pyopencl", "pyopencl",
"pyvkfft",
"open3d", "open3d",
"PyQt5", "PyQt5",
"pyqtgraph", "pyqtgraph",
"torch",
"torchvision",
) )
+46 -17
View File
@@ -1,26 +1,55 @@
# manylinux_2_28: modern glibc baseline with binary wheels for numpy/scipy/opencv. # manylinux_2_28: modern glibc baseline with binary wheels for numpy/scipy/opencv.
# Prefer this over manylinux2014 — recent NumPy requires GCC >= 10.3 and will try # Prefer this over manylinux2014 — recent NumPy requires GCC >= 10.3 and will try
# (and fail) to build from source on the older image. # (and fail) to build from source on the older image.
FROM quay.io/pypa/manylinux_2_28_x86_64:latest ARG MANYLINUX_ARCH=x86_64
FROM quay.io/pypa/manylinux_2_28_${MANYLINUX_ARCH}:latest
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 \ ENV PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_ROOT_USER_ACTION=ignore PIP_ROOT_USER_ACTION=ignore \
ORISYS_WHEELHOUSE_ROOT=/opt/orisys-wheelhouse
# Only install compile/repair tooling here. Runtime deps (numpy/scipy/opencv/numba) # Per-ABI build tooling plus a per-ABI wheelhouse of every package the release
# are declared in the wheel metadata and installed by pip when users install orisys. # flow installs at build time:
# Nuitka is configured with --nofollow-import-to for those packages. #
RUN for PYBIN in \ # * build tooling - nuitka/auditwheel etc. baked in, so the entrypoint does
/opt/python/cp310-cp310/bin \ # not have to re-resolve them from PyPI on every container start
/opt/python/cp311-cp311/bin \ # (ORISYS_UPDATE_BUILD_TOOLS=1 opts back in to that).
/opt/python/cp312-cp312/bin; do \ # * wheelhouse - `build/wheel/build.py --target pypi-linux` force-enables
echo "=== Installing build tooling into ${PYBIN} ===" && \ # verification, which creates a throwaway venv and pip-installs the built
"${PYBIN}/python" -m pip install --upgrade pip setuptools wheel && \ # wheel plus its declared runtime deps. Runtime deps are deliberately NOT
"${PYBIN}/python" -m pip install \ # installed into the image interpreter (users get them from the wheel
"nuitka>=2.0" \ # metadata), so without a wheelhouse every build re-downloaded
ordered-set \ # numpy/scipy/opencv/numba — roughly 200 MB, three times over, once per ABI.
zstandard \ # The entrypoint points pip at the wheelhouse with
auditwheel; \ # PIP_NO_INDEX/PIP_FIND_LINKS, which keeps the venv isolated and still
done # exercises real dependency resolution, just without the network.
#
# --only-binary=:all: makes a missing wheel a loud image-build failure instead of
# a slow source build inside the container.
RUN set -eux; \
for TAG in cp310-cp310 cp311-cp311 cp312-cp312; do \
PYBIN="/opt/python/${TAG}/bin"; \
echo "=== build tooling for ${TAG} ==="; \
"${PYBIN}/python" -m pip install --upgrade pip setuptools wheel; \
"${PYBIN}/python" -m pip install \
"nuitka>=2.0" \
ordered-set \
zstandard \
auditwheel; \
echo "=== wheelhouse for ${TAG} ==="; \
"${PYBIN}/python" -m pip download \
--only-binary=:all: \
--dest "${ORISYS_WHEELHOUSE_ROOT}/${TAG}" \
pip \
setuptools \
wheel \
auditwheel \
numpy \
scipy \
opencv-python \
numba; \
done; \
du -sh "${ORISYS_WHEELHOUSE_ROOT}"/*
WORKDIR /src WORKDIR /src
ENTRYPOINT ["/bin/bash", "/src/build/wheel/docker/entrypoint.sh"] ENTRYPOINT ["/bin/bash", "/src/build/wheel/docker/entrypoint.sh"]
+30 -2
View File
@@ -1,17 +1,23 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Runs inside quay.io/pypa/manylinux_2_28_x86_64 with the repo mounted at /src. # Runs inside the selected manylinux_2_28 image with the repo mounted at /src.
# Builds one Nuitka wheel per CPython ABI listed in ORISYS_PYTHON_TAGS. # Builds one Nuitka wheel per CPython ABI listed in ORISYS_PYTHON_TAGS.
set -euo pipefail set -euo pipefail
TARGET="${ORISYS_BUILD_TARGET:-pypi-linux}" TARGET="${ORISYS_BUILD_TARGET:-pypi-linux}"
# Comma-separated manylinux tags, e.g. "cp310-cp310,cp311-cp311,cp312-cp312" # Comma-separated manylinux tags, e.g. "cp310-cp310,cp311-cp311,cp312-cp312"
PYTHON_TAGS="${ORISYS_PYTHON_TAGS:-cp310-cp310,cp311-cp311,cp312-cp312}" PYTHON_TAGS="${ORISYS_PYTHON_TAGS:-cp310-cp310,cp311-cp311,cp312-cp312}"
WHEELHOUSE_ROOT="${ORISYS_WHEELHOUSE_ROOT:-/opt/orisys-wheelhouse}"
# The image already bakes in pip/setuptools/wheel/nuitka/auditwheel. Re-resolving
# them from PyPI on every container start costs a network round-trip per ABI and
# lets the running container drift from the image it was built from.
UPDATE_BUILD_TOOLS="${ORISYS_UPDATE_BUILD_TOOLS:-0}"
cd /src cd /src
echo "=== manylinux entrypoint ===" echo "=== manylinux entrypoint ==="
echo "Target: ${TARGET}" echo "Target: ${TARGET}"
echo "Python tags: ${PYTHON_TAGS}" echo "Python tags: ${PYTHON_TAGS}"
echo "Wheelhouse root: ${WHEELHOUSE_ROOT}"
uname -a uname -a
IFS=',' read -r -a TAG_ARRAY <<< "${PYTHON_TAGS}" IFS=',' read -r -a TAG_ARRAY <<< "${PYTHON_TAGS}"
@@ -40,7 +46,27 @@ for tag in "${TAG_ARRAY[@]}"; do
export PATH="${PYBIN}:${PATH}" export PATH="${PYBIN}:${PATH}"
export PYTHONPATH="/src/src${PYTHONPATH:+:${PYTHONPATH}}" export PYTHONPATH="/src/src${PYTHONPATH:+:${PYTHONPATH}}"
"${PYBIN}/python" -m pip install -q --upgrade pip setuptools wheel auditwheel # Point pip at the per-ABI wheelhouse baked into the image. The pypi-linux
# target force-enables verification, which creates a throwaway venv and
# pip-installs the built wheel plus its declared runtime deps; serving those
# from the wheelhouse keeps the venv isolated (no --system-site-packages) while
# removing a ~200 MB download per ABI. PIP_* is inherited by every pip
# subprocess, including that venv.
wheelhouse="${WHEELHOUSE_ROOT}/${tag}"
if [[ -d "${wheelhouse}" ]]; then
export PIP_NO_INDEX=1
export PIP_FIND_LINKS="${wheelhouse}"
echo "[info] pip is offline, installing from ${wheelhouse}"
else
echo "[warn] no wheelhouse at ${wheelhouse}; pip will use the network." >&2
echo "[warn] rebuild the image without -SkipImageBuild to bake it in." >&2
unset PIP_NO_INDEX PIP_FIND_LINKS || true
fi
if [[ "${UPDATE_BUILD_TOOLS}" == "1" ]]; then
echo "[info] refreshing build tooling (ORISYS_UPDATE_BUILD_TOOLS=1)"
"${PYBIN}/python" -m pip install -q --upgrade pip setuptools wheel auditwheel
fi
if "${PYBIN}/python" build/wheel/build.py --target "${TARGET}"; then if "${PYBIN}/python" build/wheel/build.py --target "${TARGET}"; then
BUILT+=("${tag}") BUILT+=("${tag}")
@@ -48,6 +74,8 @@ for tag in "${TAG_ARRAY[@]}"; do
echo "[ERROR] Build failed for ${tag}" >&2 echo "[ERROR] Build failed for ${tag}" >&2
FAILED=1 FAILED=1
fi fi
unset PIP_NO_INDEX PIP_FIND_LINKS || true
done done
echo "" echo ""
+24 -5
View File
@@ -1,21 +1,24 @@
# Build PyPI-compliant manylinux wheels for CPython 3.10/3.11/3.12 using Docker Desktop. # Build x86_64 or aarch64 manylinux wheels using Docker Desktop.
# #
# Usage (from repo root, PowerShell): # Usage (from repo root, PowerShell):
# .\build\wheel\docker\run.ps1 # .\build\wheel\docker\run.ps1
# .\build\build_manylinux_docker.ps1 # compatibility wrapper # .\build\wheel\docker\run.ps1 -Architecture aarch64 -PythonVersions 3.10
# #
# From cmd.exe: # From cmd.exe:
# powershell -ExecutionPolicy Bypass -File .\build\build_manylinux_docker.ps1 # powershell -ExecutionPolicy Bypass -File .\build\wheel\docker\run.ps1
param( param(
[ValidateSet("pypi-linux", "internal")] [ValidateSet("pypi-linux", "internal")]
[string]$Target = "pypi-linux", [string]$Target = "pypi-linux",
[string]$PythonVersions = "3.10,3.11,3.12", [string]$PythonVersions = "3.10,3.11,3.12",
[ValidateSet("x86_64", "aarch64")]
[string]$Architecture = "x86_64",
[switch]$SkipImageBuild, [switch]$SkipImageBuild,
[string]$ImageName = "orisys-manylinux" [string]$ImageName = ""
) )
$ErrorActionPreference = "Stop" $ErrorActionPreference = "Stop"
$Architecture = $Architecture.ToLowerInvariant()
function Convert-ToManylinuxTag { function Convert-ToManylinuxTag {
param([string]$Version) param([string]$Version)
@@ -45,11 +48,17 @@ if ($tags.Count -eq 0) {
throw "No Python versions specified." throw "No Python versions specified."
} }
$pythonTags = [string]::Join(",", $tags) $pythonTags = [string]::Join(",", $tags)
$dockerArchitecture = if ($Architecture -eq "aarch64") { "arm64" } else { "amd64" }
$dockerPlatform = "linux/$dockerArchitecture"
if ([string]::IsNullOrWhiteSpace($ImageName)) {
$ImageName = if ($Architecture -eq "aarch64") { "orisys-manylinux-aarch64" } else { "orisys-manylinux" }
}
Write-Host "=== Orisys manylinux Docker build ===" Write-Host "=== Orisys manylinux Docker build ==="
Write-Host "Repo: $RepoRoot" Write-Host "Repo: $RepoRoot"
Write-Host "Target: $Target" Write-Host "Target: $Target"
Write-Host "Image: $ImageName" Write-Host "Image: $ImageName"
Write-Host "Architecture: $Architecture ($dockerPlatform)"
Write-Host "Python tags: $pythonTags" Write-Host "Python tags: $pythonTags"
# Docker Desktop must be in Linux container mode for manylinux. # Docker Desktop must be in Linux container mode for manylinux.
@@ -71,15 +80,25 @@ $dockerfile = "build/wheel/docker/Dockerfile.manylinux"
if (-not $SkipImageBuild) { if (-not $SkipImageBuild) {
Write-Host "`n=== Building Docker image ===" Write-Host "`n=== Building Docker image ==="
docker build -f $dockerfile -t $ImageName . docker build --platform $dockerPlatform --build-arg "MANYLINUX_ARCH=$Architecture" -f $dockerfile -t $ImageName .
if ($LASTEXITCODE -ne 0) { if ($LASTEXITCODE -ne 0) {
throw "docker build failed" throw "docker build failed"
} }
} }
# A custom or reused image must contain the requested architecture.
$imageArchitecture = docker image inspect --format "{{.Architecture}}" $ImageName
if ($LASTEXITCODE -ne 0) {
throw "Cannot inspect image '$ImageName'. Rebuild without -SkipImageBuild."
}
if ($imageArchitecture -ne $dockerArchitecture) {
throw "Image '$ImageName' architecture is '$imageArchitecture'; expected '$dockerArchitecture'. Rebuild without -SkipImageBuild."
}
Write-Host "`n=== Running manylinux container ===" Write-Host "`n=== Running manylinux container ==="
# Mount the repo so artifacts land under dist/repaired and SDK/<version>/ on the host. # Mount the repo so artifacts land under dist/repaired and SDK/<version>/ on the host.
docker run --rm ` docker run --rm `
--platform $dockerPlatform `
-e "ORISYS_BUILD_TARGET=$Target" ` -e "ORISYS_BUILD_TARGET=$Target" `
-e "ORISYS_PYTHON_TAGS=$pythonTags" ` -e "ORISYS_PYTHON_TAGS=$pythonTags" `
-v "${RepoRoot}:/src" ` -v "${RepoRoot}:/src" `
+87 -7
View File
@@ -16,9 +16,79 @@ from common.paths import sdk_dir
# PyPI rejects bare tags like "manylinux_x86_64"; require a glibc ABI version. # PyPI rejects bare tags like "manylinux_x86_64"; require a glibc ABI version.
_MANYLINUX_TAG_RE = re.compile( _MANYLINUX_TAG_RE = re.compile(
r"(?:manylinux2014|manylinux_2_(?:17|24|27|28|31|34|35))_x86_64" r"(?:manylinux2014|manylinux_2_(?:17|24|27|28|31|34|35))_(?:x86_64|aarch64)(?=\.|$)"
) )
# Source of the PyPI project description (see read_long_description).
PYPI_README_RELPATH = Path("docs") / "pypi_readme.md"
def _license_text(project: dict) -> str | None:
"""Extract free-text license from either pyproject license form."""
declared = project.get("license")
if isinstance(declared, str):
return declared
if isinstance(declared, dict):
text = declared.get("text")
if isinstance(text, str):
return text
return None
def _person_strings(entries) -> list[str]:
"""Render PEP 621 author/maintainer tables as ``Name <email>`` strings."""
rendered: list[str] = []
for entry in entries or []:
if not isinstance(entry, dict):
continue
name = (entry.get("name") or "").strip()
email = (entry.get("email") or "").strip()
if name and email:
rendered.append(f"{name} <{email}>")
elif name or email:
rendered.append(name or email)
return rendered
def metadata_kwargs(project: dict) -> dict:
"""Translate a pyproject ``[project]`` table into ``setup()`` keywords.
The Nuitka wheel is built by a generated ``setup.py`` outside the normal
setuptools pipeline, so every PyPI-visible field has to be passed through
explicitly. Omitting them is what left the early PyPI releases with no
``Requires-Python``, classifiers, project URLs or license on the project page.
"""
authors = _person_strings(project.get("authors"))
maintainers = _person_strings(project.get("maintainers"))
candidates = {
"description": project.get("description"),
"python_requires": project.get("requires-python"),
"classifiers": list(project.get("classifiers") or []),
"project_urls": dict(project.get("urls") or {}),
"license": _license_text(project),
"author": authors[0] if authors else None,
"author_email": None,
"maintainer": maintainers[0] if maintainers else None,
}
return {key: value for key, value in candidates.items() if value}
def read_long_description(repo_root: Path) -> str | None:
"""Return the PyPI long description, or ``None`` when it is missing.
Deliberately not the repository ``README.md``: that file documents the
closed-source build/publishing layout, which should not end up on the PyPI
project page.
"""
path = Path(repo_root) / PYPI_README_RELPATH
if not path.is_file():
print(
f"[WARN] {PYPI_README_RELPATH.as_posix()} not found; the wheel will be "
"published without a long description."
)
return None
return path.read_text(encoding="utf-8")
def write_wheel_setup( def write_wheel_setup(
wheel_dir: Path, wheel_dir: Path,
@@ -27,7 +97,17 @@ def write_wheel_setup(
package_data: dict[str, list[str]], package_data: dict[str, list[str]],
install_requires: list[str], install_requires: list[str],
extras_require: dict[str, list[str]], extras_require: dict[str, list[str]],
project: dict | None = None,
long_description: str | None = None,
) -> Path: ) -> Path:
metadata_lines = ""
if project:
for key, value in sorted(metadata_kwargs(project).items()):
metadata_lines += f" {key}={value!r},\n"
if long_description:
metadata_lines += " long_description=_LONG_DESCRIPTION,\n"
metadata_lines += ' long_description_content_type="text/markdown",\n'
setup_py_path = wheel_dir / "setup.py" setup_py_path = wheel_dir / "setup.py"
setup_content = f""" setup_content = f"""
from setuptools import setup, Distribution from setuptools import setup, Distribution
@@ -36,12 +116,12 @@ class BinaryDistribution(Distribution):
def has_ext_modules(self): def has_ext_modules(self):
return True return True
{'_LONG_DESCRIPTION = ' + repr(long_description) if long_description else ''}
setup( setup(
name="orisys", name="orisys",
version="{version}", version="{version}",
description="Orisys - Optical Tactile Sensor SDK (Nuitka Compiled)", {metadata_lines} packages=['orisys', 'orisys.examples', 'orisys_blend'],
author="Orisys",
packages=['orisys', 'orisys.examples', 'orisys_blend'],
package_dir={{'': 'wheel_root'}}, package_dir={{'': 'wheel_root'}},
package_data={package_data}, package_data={package_data},
include_package_data=True, include_package_data=True,
@@ -75,7 +155,7 @@ def _ensure_auditwheel() -> None:
def is_compliant_manylinux_wheel(wheel_path: Path) -> bool: def is_compliant_manylinux_wheel(wheel_path: Path) -> bool:
"""True if the wheel filename uses a PyPI-accepted manylinux x86_64 tag.""" """True if the wheel filename uses a supported manylinux x86_64 or aarch64 tag."""
return bool(_MANYLINUX_TAG_RE.search(wheel_path.name)) return bool(_MANYLINUX_TAG_RE.search(wheel_path.name))
@@ -120,8 +200,8 @@ def repair_linux_wheel(wheel_path: Path, repair_dir: Path) -> Path:
names = ", ".join(path.name for path in repaired) names = ", ".join(path.name for path in repaired)
raise RuntimeError( raise RuntimeError(
"auditwheel repair did not produce a PyPI-compliant manylinux tag. " "auditwheel repair did not produce a PyPI-compliant manylinux tag. "
f"Got: {names}. Rebuild inside quay.io/pypa/manylinux_2_28_x86_64 " f"Got: {names}. Rebuild inside the manylinux_2_28 image for the "
"(or manylinux2014_x86_64) instead of a generic Linux host." "target architecture instead of a generic Linux host."
) )
final = compliant[0] final = compliant[0]
Binary file not shown.
+147
View File
@@ -0,0 +1,147 @@
# Public API surface audit (2026-09-30)
A snapshot of what the SDK declares as public, what is documented, and where the gaps are.
Produced by [`tools/api_audit.py`](../tools/api_audit.py), which is static and read-only — it
never imports the package, so it costs no numpy/cv2 import and cannot be affected by the
environment. Re-run it after any API change:
```powershell
cd D:\orisys_ws\orisys-sdk
python tools/api_audit.py # text report
python tools/api_audit.py --json # machine-readable
python tools/api_audit.py --quiet # headline numbers
```
## 1. Headline
| Measure | Value |
|---|---|
| Public classes / methods / functions | **393** (53 / 214 / 126) |
| With a docstring | **256 (65 %)** |
| Fully annotated signature (params + return) | **125 (32 %)** |
| Docstring documents its return value | **38** |
Docstring conventions actually in use:
| Convention | Count |
|---|---|
| plain prose (no structured sections) | 243 |
| none | 137 |
| Google (`Args:` / `Returns:`) | 9 |
| NumPy (`Parameters` + `---`) | 4 |
**Reading:** the dominant convention is not a convention. Structured sections are absent from
almost every docstring, and two thirds of signatures are missing type information. The hand-written
reference compensates in the places it covers; a generated reference would not.
## 2. The declared public surface
The root `__all__` (`src/orisys/__init__.py`) declares 18 names. Sub-surfaces declare a further 88.
### 2.1 Root exports — 9 of 18 are absent from `docs/api.md`
| Documented | Not mentioned anywhere in `docs/api.md` |
|---|---|
| `Sensor`, `util`, `finger`, `signal_processing`, `capture`, `list_config`, `export_config`, `export_sensor_bundle`, `export_sensor_bundles` | **`StreamPipeline`**, **`FrameSnapshot`**, **`PipelineStats`**, **`StreamRecorder`**, **`mosaic_videos`**, **`FFCFrameProcessor`**, `CompoundEyeSensor`, `CameraSensor`, `calibration` |
`StreamPipeline` is the engine behind the published "5 × 30 FPS on one desktop, 4.2× headroom"
claim and is used by `examples/multi_stream.py`. `StreamRecorder` and `mosaic_videos` are the
offline-recording path. None of them has a Markdown reference anywhere in the repository.
### 2.2 Sub-surfaces with no coverage in `api.md` at all
| Module | Exports | Documented | Note |
|---|---|---|---|
| `orisys.unroll` | 18 | 0 | UV domain, remap, geometry — the finger pipeline's core maths |
| `orisys.capture` | 17 | 0 | capture sources, workers, reconnect policy, camera discovery |
| `orisys.hhd` + `hhd.solvers` | 7 | 0 | **no docstrings either** — see §3 |
| `orisys.pipeline` | 3 | 0 | `StreamPipeline` etc. |
| `orisys.export` | 2 | 0 | `StreamRecorder`, `mosaic_videos` |
### 2.3 Exports with no docstring — 20 of 88
Includes `CaptureWorker`, `pick_default_camera`, `available_blend_backends`,
`compute_final_crop_rect`, `coverage_ratio`, `largest_fully_covered_rect`,
`match_bgr_brightness`, `barycentric_grid`, `face_uvs_from_loop_data`, all four `orisys.hhd`
exports, and named constants (`DEFAULT_LAPTOP_WEBCAM_PATTERNS`, `MAX_SIMULTANEOUS_CAPTURES`,
`DEFAULT_FINGER_EXPORT_DIR`, `DEFAULT_FINGER_MAP_CACHE`, `BLEND_MODES`).
## 3. Coverage by module
The extreme cases, which are where effort pays off:
| Module | Symbols | Docstrings | Annotated | Comment |
|---|---|---|---|---|
| `orisys.finger.runtime` | 31 | **35 %** | **3 %** | `FingerRuntime` is the documented entry point for finger users, yet its implementation module is the thinnest in the package |
| `orisys.hhd.timing` | 4 | **0 %** | 0 % | |
| `orisys.stitch.feature` | 10 | 20 % | 0 % | |
| `orisys.stitch.tiles` | 5 | 20 % | 0 % | |
| `orisys.stitch.api` | 10 | 30 % | 10 % | |
| `orisys.frame_processors` | 21 | 48 % | **0 %** | |
| `orisys.capture.source` | 32 | 50 % | 91 % | annotations good, prose thin |
| `orisys.sensor` | 21 | 90 % | **38 %** | the most-documented module, still missing annotations |
| `orisys.configs` | 13 | 100 % | 69 % | |
| `orisys.unroll.precompute`, `domain` | 6 / 6 | 100 % | 0 % | documented but untyped |
Annotated and documented do not move together: `capture.source` is 91 % annotated but 50 %
documented; `hhd.domain` is 100 % documented but 0 % annotated.
## 4. Used by examples and tests, absent from the reference — 74 names
These are APIs users demonstrably reach for and the reference never explains. The highest-traffic
group is the runtime/typing surface:
`StreamPipeline`, `FrameSnapshot`, `PipelineStats`, `StreamRecorder`, `mosaic_videos`,
`FFCFrameProcessor`, `UVDomain`, `UVRemap`, `UVSurfaceMap`, `UVSurfaceMesh`, `UVMeshViewer`,
`Tracker`, `Contact3D`, `CalSample`, `CalibrationError`, `StitchStatus`, `CameraInfo`,
`CameraFilter`, `CaptureSource`, `CaptureConfig`, `CaptureHealth`, `ReconnectPolicy`,
`PoissonSolver`, `nHHD`
Plus the finger/geometry helpers (`build_uv_surface_mesh`, `compute_uv_surface_frames`,
`lift_uv_flow_to_3d`, `flow_magnitude_vertex_colors`, `scalar_map_vertex_colors`,
`sample_field_at_mesh_vertices`, `mesh_uv_to_world_scale`, `resolve_export_paths`,
`resolve_finger_geometry`, `load_finger_export_bundle`, `init_uv_mesh_viewer`,
`update_uv_mesh_viewer`), the capture helpers (`open_capture`, `normalize_source`,
`list_cameras`, `pick_default_camera`), the calibration helpers (`check`, `build`, `parse`,
`load`, `calibrate_normal_force`, `calibrate_shear_force`, `calibrate_shear_components`),
and the config resolvers (`load_config`, `list_sensor_bundles`, `resolve_sensor_config_path`,
`resolve_sensor_resource`, `resolve_sensor_asset_dir`, `pipeline_from_sensor_json`).
## 5. Public classes exported nowhere — 21
Defined, carrying no name-mangling underscore, but in no `__all__` and not reachable as a declared
API: `BlendBackend`, `CalSample`, `Calibration`, `CalibrationError`, `CausalMedianFilter`,
`ChannelCurve`, `CheckResult`, `CompoundEyeFrameProcessor`, `Contact3D`,
`ForceChannelCalibration`, `FrameUnwarper`, `LowpassFilter`, `MaskedDomain`,
`MaskedPoissonConfig`, `NumbaBlendBackend`, `NumpyBlendBackend`, `SampleRateEstimator`,
`SensorBundle`, `SingleCameraFrameProcessor`, `Tracker`, `VideoCaptureLike`.
Two of these are the clearest signals of a boundary that was never decided:
- **`LowpassFilter`** is documented in `docs/api.md` as `orisys.signal_processing.LowpassFilter`
and is imported by examples — but it appears in neither the submodule's `__all__` nor the root's.
It is public by documentation and private by declaration.
- **`Calibration`** / `CalibrationError` / `CalSample` / `CheckResult` are the entire force
calibration object model, and `calibration.check()` / `calibration.build()` are used directly by
the test suite. The `orisys.calibration` package is exported from the root, but none of its
public names are declared or documented.
Each of the 21 needs one decision: promote into the relevant `__all__`, or rename with a leading
underscore to mark it internal.
## 6. What this implies
1. **Fix the declaration boundary first.** Nine root exports have no reference and 21 classes sit
in the gap between documented and declared. Publishing a reference on top of an undecided
public surface multiplies the confusion rather than resolving it.
2. **Pick one docstring convention before generating anything.** With 243 prose docstrings, 137
absent, and 13 structured, a generator's output would be thin and inconsistent — the tool
renders what is there.
3. **Annotations are the larger gap.** 32 % complete signatures means generated parameter tables
would be mostly bare. This is mechanical work and it improves IDE help, not only docs.
4. **The omissions are not random.** `StreamPipeline`, `capture` and `unroll` are the *newest*
subsystems, and they are the ones with no coverage. The reference drifted toward the parts that
existed when it was written.
5. **`orisys.hhd` is documented nowhere and has no docstrings**, yet is exported from the root
package. Either document it or stop exporting it — leaving it in `__all__` promises support that
does not exist.
+21 -2
View File
@@ -17,7 +17,7 @@ description: Orisys SDK Python API 中文参考
* **vid_src** (`int | str`, 必填): 传感器 ID、相机编号或视频路径。默认为 `0`。 * **vid_src** (`int | str`, 必填): 传感器 ID、相机编号或视频路径。默认为 `0`。
* **config_name** (`str`,可选): 传感器配置文件名称或 `json` 文件路径,用于加载对应的参数配置;例如 `box` 或 `./config/box.json`。 * **config_name** (`str`,可选): 传感器配置文件名称或 `json` 文件路径,用于加载对应的参数配置;例如 `box` 或 `./config/box.json`。
* **cal_path** (`str`,可选,已弃用): 该参数仅为兼容旧代码保留,当前版本中不再使用,传入后也会被忽略。 * **cal_path** (`str`,可选): 显式的 `force_calibration.json` 路径;传入后优先于 `sensors/<type>/force_calibration.json`。不传时按 `config_name` 自动加载产品包中的标定文件。
* **backend** (`str`,可选): 光流场解耦计算后端,默认为 `auto`,可选`cupy`,`opencl`,`cpu`。 * **backend** (`str`,可选): 光流场解耦计算后端,默认为 `auto`,可选`cupy`,`opencl`,`cpu`。
* **verbose** (`bool`,可选): 是否输出初始化及运行过程中的详细日志信息,默认为 `False`。 * **verbose** (`bool`,可选): 是否输出初始化及运行过程中的详细日志信息,默认为 `False`。
@@ -205,7 +205,6 @@ sensor.configure(lowpass_cutoff_hz=5.0, lowpass_sample_rate_hz=30.0)
* **lowpass_sample_rate_hz** (`float`,可选): 力信号采样率,默认 `30.0`。 * **lowpass_sample_rate_hz** (`float`,可选): 力信号采样率,默认 `30.0`。
* **lowpass_order** (`int`,可选): Butterworth 滤波器阶数,默认 `2`。 * **lowpass_order** (`int`,可选): Butterworth 滤波器阶数,默认 `2`。
* **lowpass_enabled** (`bool`,可选): 显式开启或关闭低通滤波。 * **lowpass_enabled** (`bool`,可选): 显式开启或关闭低通滤波。
* **hhd_downsample** (`int`,可选): HHD 解算下采样倍数,需为 `>= 1` 的整数,配置默认 `3`。
#### 读取结果 #### 读取结果
@@ -577,6 +576,26 @@ finally:
* 导出内置的`config_name`配置参数字典至`path` * 导出内置的`config_name`配置参数字典至`path`
### `export_sensor_bundle` / `export_sensor_bundles` 方法
#### 描述
将已安装的 `sensors/<name>/` 传感器包复制到目标目录,供下游应用(如桌面 App)做本地可编辑配置,无需硬编码 wheel 路径。解析顺序与 `resolve_sensor_bundle` 相同(含 `importlib.resources` 打包布局)。
#### 输入参数
* **name** / **names**: 单个名字,或名字列表;`export_sensor_bundles` 省略 `names` 时导出所有可解析的产品包(`box` / `mini` / `finger`)
* **dest_dir**: 目标根目录;实际写入 `dest_dir/<name>/`
* **include_assets**: 默认 `False`,仅复制 `sensor.json` 以及存在的 `force_calibration.json` / `manifest.json`;为 `True` 时复制完整目录树(含 `assets/`)
* **overwrite**: 默认 `True`;为 `False` 且目标已存在时抛出 `FileExistsError`
#### 返回
* `Path` 或 `list[Path]`:导出后的包目录路径
#### CLI
```bash
python -m orisys.export_sensors --dest ./out/sensors --names box,mini
python -m orisys.export_sensors --dest ./out/sensors --include-assets
```
## 常见问题解答 (FAQ) ## 常见问题解答 (FAQ)
### Q1:为什么必须先调用`get_img()`,再调用`compute_deformation()`? ### Q1:为什么必须先调用`get_img()`,再调用`compute_deformation()`?
A:`compute_deformation()`依赖最近一次获取并拼接的图像数据进行形变与力计算。 A:`compute_deformation()`依赖最近一次获取并拼接的图像数据进行形变与力计算。
+74
View File
@@ -0,0 +1,74 @@
# Code review: the automated part
Four gates existed before this document (the test suites, `apps/studio/tools/ui_check.py`, the
real-GL smoke tests, the binding-purity test) and none of them is a linter. This adds the missing
*static* layer, deliberately small.
## Why it exists, in one measured fact
Until now the three repos had **no Ruff configuration at all**. `ruff check .` therefore ran
whatever the installed version enables by default — Ruff 0.16 enables a very broad set
(`UP`/`RUF`/`PLR`/`S`/`TRY`/`PERF`/`FURB`/`YTT`/`ASYNC`…) — so the same tree produced **1038 /
438 / 117** findings in `orisys-app` / `orisys-sdk` / `orisys-sensor-design`, and a different
Ruff version produced different ones. A gate built on that is not a gate. `ruff.toml` now pins it.
## What is gated, and what is deliberately not
| rule | meaning | gated? | why |
|---|---|---|---|
| `BLE001` | broad `except Exception` | **no** | A broad catch is the *right* choice around device I/O, GL teardown and camera probing. This codebase already records that choice inline as `# noqa: BLE001 - <reason>`. Gating it would produce ~100 findings whose only correct fix is "add a reasoned suppression". |
| `S110`, `S112` | handler is only `pass` / `continue` | **yes** | The complaint is not the breadth of the catch, it is that the failure becomes **invisible**. Silence is acceptable only with a reasoned suppression or a handler that reports/re-raises. |
| `E722` | bare `except:` | **yes** | Strictly worse than `except Exception:` — it also swallows `KeyboardInterrupt`/`SystemExit`. |
| `F` | unused import, redefinition, undefined name, unused variable | **yes** | Real defects: `F821`/`F822` are latent `NameError`s, `F811` can be a shadowed definition. |
| `B023` | closure over a loop variable | **yes** | Late binding silently produces wrong geometry in the headless Blender scripts. |
| `PLW1510` | `subprocess.run` without `check` | **yes** | A swallowed tool failure. |
| `RUF100` | unused suppression | reported, **never auto-fixed** | The `- reason` in a `noqa` comment is documentation. A machine must not delete it. Triage by hand. |
| `UP`, `I`, `PIE`, `RET501`, `PLR1711`, `SIM114`, `RUF022`, `RUF023` | mechanical modernisation | in the lint config, **not** in the gate | Applied once by `--fix`; gating them would only churn AI-authored diffs. |
`E402` (module import not at top of file) is **selected but not gated**. `import` after
`os.environ[...]` is deliberate in this codebase — the environment must be set before Qt/cv2/the SDK
load — so gating it would be pure noise. But it is the only rule that would catch an *accidental*
import moved above that setup, so it stays visible. When you touch one of those files, record the
intent as `# noqa: E402 - <reason>`; `E701`/`E702` are style, which `ruff format` owns. A gate that
cries wolf gets ignored, which is worse than no gate.
## The three commands
```powershell
# 1. The agent loop, after every edit — mechanical, no review needed.
ruff check <changed files> --fix # never --unsafe-fixes
ruff check <changed files> --output-format json # what remains, machine-readable
# 2. The gate before a change lands. Changed files only, machine-readable, non-zero on findings.
python tools/lint_gate.py # vs HEAD
python tools/lint_gate.py --base origin/main
python tools/lint_gate.py --all # gate rules over the whole repo
python tools/lint_gate.py --json
# 3. Formatting, changed files only — never a whole-tree rewrite mid-diff.
ruff format <changed files>
```
`$RUFF` may point at a specific ruff binary; otherwise `tools/lint_gate.py` uses `PATH` or the one
beside the interpreter. **Pin the version**: the gate prints the version it ran, and findings differ
between versions, so a runner must install the same one the developers use (`ruff==0.16.9`).
## What this found when it was introduced
Filtering to rules that can indicate a defect produced ~18 candidates rather than style noise —
which is the argument for the narrow set:
- six latent `NameError`s (`np`, `sys`, `Callable`, `FrameIdentity`, `tessellate_polygon`, and an
`__all__` entry naming a function that does not exist) — none caught by any test;
- a **duplicated** function definition that silently shadowed the first;
- `lower == lower` / `upper == upper` in a range check, and the same pattern inside a **test
assertion that could never fail**;
- 13 closures over loop variables in the headless Blender/patch scripts;
- two `subprocess.run` calls whose failure was silently ignored.
## Follow-up deliberately not done yet
`PLR0124` ("name compared with itself") is not in `select`. Eleven of its sites are the deliberate
`value != value` NaN idiom and must become `math.isnan(...)` (or carry a reasoned suppression)
before the rule can be gated — otherwise it is noise. Once those are converted, add `PLR0124` to
`select` and to `GATE_SELECT` in `tools/lint_gate.py`.
+120
View File
@@ -0,0 +1,120 @@
# 滤波:三级、预设与可追溯性
`json.load` 不支持注释(见 `src/orisys/configs.py`),所以"建议的滤波"**不写在 JSON 注释里**,
而是写成**代码里的预设**(`src/orisys/sensor.py::FILTER_PRESETS`)+ 这份文档 +
`examples/filters/` 里可直接复制的配置。预设是可执行、可回显、可审计的;注释不是。
---
## 1. 三级滤波,位置不同、性质不同
| 级 | 配置键 | 作用位置 | 性质 |
|---|---|---|---|
| 光流时间 EMA | `flow_ema_alpha` | `compute_flow()` 内,**HHD 之前** | **改变测量**:改 `vnormal`/接触统计与 `_force_raw` 的值。两个 `alpha` 不同的录制**不可比** |
| 力中值 | `force_median_window` | `_apply_force_filter()`:raw → 中值 → 低通 → 标定 | **读数滤波**:删除单点尖峰(因果滑动中值) |
| 力低通 | `force_lowpass_cutoff_hz` / `_order` / `_sample_rate_hz` | 同上(中值之后) | **读数滤波**:因果 Butterworth IIR(`sosfilt` + 持久状态) |
因此 `filters_state()` 把它们分成两组:`measurement`(测量身份)与 `readout`(读数滤波)。
标定/可比性检查应比对 `measurement`,记录里两者都记。
## 2. 取值规则(自描述,无隐藏覆盖)
- **每个级由自己的值开关**:`flow_ema_alpha >= 1`、`force_median_window <= 1`、
`force_lowpass_cutoff_hz <= 0` 都是"关"。
- **配置没写的键 = 关**:缺省不是"我们觉得合理的那档",而是"不滤波"。因此**没有 `filters`
段的配置文件什么都不滤**,出厂 bundle 也把三个值显式写成全关 —— 想让哪一级跑起来,
必须在文件里写出来(数值或 `preset`)。这样"文件说的"和"设备跑的"才总是一致。
- **非法值 → 关**,绝不回落到"默认开"。
- `filters.enabled` 是**废弃别名**,显式 `false` 表示"三级全关";没有任何开关能覆盖已写的值。
- `filters.preset` 可引用下面的预设;**显式键优先于预设值**(可从预设出发只改一个旋钮)。
- 任何时刻的**真值**是 `Sensor.filters_state()`,不是配置文件 —— 运行时 `configure()` 可以改它。
## 3. 预设(`FILTER_PRESETS`)
| 预设 | `flow_ema_alpha` | `force_median_window` | `force_lowpass_cutoff_hz` | 延迟 @30fps | 用途 |
|---|---|---|---|---|---|
| `off` | 1.0 | 1 | 0.0 | 0 帧 | 原始读数;文件说什么就跑什么 |
| `low_latency` | 1.0 | 3 | 0.0 | ≈1 帧(33 ms) | 延迟敏感,只去尖峰 |
| `balanced` | 0.5 | 3 | 0.0 | ≈2 帧(67 ms) | 建议起步档:EMA 干掉力级滤波够不到的光流抖动 |
| `low_noise` | 0.3 | 3 | 10.0 | ≈3 帧(100 ms) | 底噪优先,读数再低通一道 |
延迟换算 @60fps 减半;数字来自 `_perf/`(该一次性目录已从 `orisys-sdk` 移除:幸存的脚本与片源
迁到 `D:\orisys_ws\docs\performance\perf-measurements\`,其余已永久丢失;对照表见
`D:\orisys_ws\docs\performance\orisys-sdk-perf-plan.md`,其中仍保留全部实测数字)。
≥1 帧的延迟与相位差是**因果滤波的代价**:旧显示端曾用零相位 `filtfilt`,那需要"未来",
无法在采集里做,也让"显示 == 记录"不可能——所以现在只有一条因果链。
## 4. 怎么用
```jsonc
// sensors/mini/sensor.json(片段;完整可复制版本见 examples/filters/)
"filters": { "preset": "balanced" }
```
```python
# 或运行时(UI 的齿轮就走这条路;不重连相机)
sensor.configure(flow_ema_alpha=0.5, force_median_window=3, lowpass_cutoff_hz=0.0)
print(sensor.filters_state()) # 实际生效值 + preset + applied 标志
```
UI 想列出可选档位:`from orisys.sensor import available_filter_presets`。
## 5. 两个 buffer:滤波前 / 滤波后
```python
t = sensor.force_triplets()
t["filtered"]["raw"], t["filtered"]["newton"] # 记录与显示用的值(同一套)
t["unfiltered"]["raw"], t["unfiltered"]["newton"] # 中值/低通之前;同帧、同标定系数
```
- `filtered.raw` = 链的输出(标定前),`unfiltered.raw` = 设备计数;
- 两个 `newton` 用同一份 `force_calibration.json`,差异**只**来自滤波链;
- 实测:注入 `10,10,10,999,10,10`(中值窗口 3)→ `filtered.raw` 在尖峰帧为 `10.0`、
`unfiltered.raw` 为 `999.0`。
## 6. 可追溯性清单
1. 录制元数据写 `sensor.filters_state()` 的**生效值**(不是配置文件内容),并含 `preset` 与
`"source": "sensor.filters_state"` + `sdk_version`。
2. 界面显示"实际生效"而不是"我填的值";两者不一致时以实际为准并提示。
3. 记录里主列 = 滤波后(与界面一致),未滤波列另存;列结构与格式版本可区分。
4. 标定/可比性指纹用 `measurement` 组;`readout` 组变化只记录、不拦截。
5. 双份数据在同一帧、同一份标定系数下产生,差异可完全归因于滤波链。
## 7. 指纹:标定文件里怎么记(已实现)
`orisys.calibration` 负责把滤波设置写进标定文件的 `meta`,并在读回时还原:
```python
from orisys.calibration import filters_fingerprint, filter_mismatches, pipeline_from_sensor_json
pipeline = pipeline_from_sensor_json(config_path, filters_state=sensor.filters_state())
# {"stitch_resolution": …, "dis_mode": …, "hhd_downsample": …, "config_ref": …,
# "filters": {"measurement": {"flow_ema_alpha": 0.5},
# "readout": {"force_median_window": 3, "force_lowpass_cutoff_hz": 0.0},
# "preset": "balanced", "source": "sensor.filters_state"}}
```
写入的文件长这样(`meta` 里同时保留扁平键供旧读者按名取用):
```json
"meta": {
"filters": {
"measurement": {"flow_ema_alpha": 0.5},
"readout": {"force_median_window": 3, "force_lowpass_cutoff_hz": 0.0},
"preset": "balanced",
"source": "sensor.filters_state"
},
"flow_ema_alpha": 0.5,
"force_median_window": 3
}
```
规则:
- **生效值优先**:传了 `filters_state=` 就用它;没传才读文件的 `filters` 段,并标成
`"source": "sensor.json.filters"`(那是"意图",不一定等于设备正在做的事)。
- **不猜**:来源没写的键**不出现**在指纹里。`{}` 表示"没有任何记录",绝不表示"大概是关的"。
旧文档写的扁平键(`meta.flow_ema_alpha` …)在读回时会被折进对应的组,因此仍可比。
- **`enabled: false`** 只作为"全部关"的废弃别名保留(`flow_ema_alpha=1`、中值 `1`、低通 `0`)。
- **分组判读**:`calibration.check(doc, pipeline=…)` 把两组的差异分开报告 —— measurement 组的
差异进 `warnings`(措辞含 `not comparable`,即"两份记录不可比"),readout 组的差异进
`notes`(只记录,不拦截)。`CheckResult.notes` 是为此新增的字段。
+77
View File
@@ -0,0 +1,77 @@
---
title: Get started with Orisys SDK development
description: Set up a source checkout, run a first test, and find the right example.
---
# Get started with Orisys SDK development
This guide is for contributors working from the SDK source checkout. Obtain the
repository URL from your team, clone it, and change into the `orisys-sdk` directory.
If you only need to use a published wheel, follow the
[Chinese installation guide](get_started.md). Run the commands below from the
repository root.
## First working loop
Use Python 3.10, 3.11, or 3.12. The existing README uses conda; an isolated
environment keeps another installed `orisys` package from shadowing this checkout.
```bash
conda create -n orisys-dev python=3.10
conda activate orisys-dev
python -m pip install -e . pytest
python -c "import orisys; print(orisys.__version__)"
python -m pytest -m "not slow and not hardware" tests/test_calibration.py
```
The editable install makes changes under `src/orisys/` visible without rebuilding a
wheel. `pytest.ini` adds `src/` to the test import path. Before debugging a strange
import failure, check that `python` and `python -m pip` point to the same environment.
The full non-hardware suite is:
```bash
python -m pytest -m "not slow and not hardware"
```
Some tests need a recorded clip. `tests/conftest.py` looks for a repository sample or
the path in `ORISYS_TEST_VIDEO`; a missing clip can skip those tests. Inspect the test
summary rather than treating a run with unexpected skips as coverage.
## Run a pipeline
The examples accept a camera index or, where documented, a video path. Start with
the example matching your sensor. A camera example needs the device and its bundle;
the test command above does not need physical hardware.
```bash
python examples/basic_pipeline.py -v 0 -c box
python examples/basic_pipeline.py -v 0 -c mini
python examples/finger_pipeline.py --camera_id 0 --config finger
```
For a video-only first run, see the exact arguments in
[examples/README.md](../examples/README.md). GUI examples need the optional GUI
dependencies (`python -m pip install -e ".[gui]"`). GPU support is optional; the
declared extras are `cuda12` and `cuda13` in [pyproject.toml](../pyproject.toml).
Start on CPU unless your task requires a GPU backend.
## Find the owner of a change
| What you are changing | Start here |
| --- | --- |
| SDK API or algorithms | `src/orisys/`, then a focused test in `tests/` |
| Product configuration and shipped assets | `sensors/<product>/`; see [sensor bundles](sensor-bundles.md) |
| Example behavior | `examples/` and [examples/README.md](../examples/README.md) |
| Build or release tooling | `build/` and [build/README.md](../build/README.md) |
| Studio UI | The separate `orisys-app` repository |
`config/` holds legacy configurations. New product-bundle work belongs under
`sensors/`. For a physical device's calibration, use a user-selected calibration
directory and `device_id`; the [SDK README](../README.md#configuration) describes
the path and validation contract. Do not commit individual sensor calibration data
to the generic product bundle by accident.
After a change, run the focused tests first, then the relevant non-hardware suite.
If a change requires a camera, record the product, configuration, input, and observed
result in your handoff. See [publishing](publishing.en.md) when preparing a release;
ordinary development does not require a wheel build or a PyPI account.
+55 -79
View File
@@ -5,10 +5,8 @@ description: 安装并运行 Orisys SDK 的基础步骤
# 快速开始 # 快速开始
版本:0.5.0
## 简介 ## 简介
本文件介绍如何上手和运行叠动视触觉传感器配套的开发套件。本套件是配合叠动视触觉传感器进行图像处理,计算形变、力以及接触信息。它可以从摄像头或视频文件中读取图像,处理后打印和展示处理后信息。 本文件介绍如何上手和运行叠动视触觉传感器配套的SDK,进行图像处理,计算形变、力以及接触信息,接受从摄像头或视频文件中读取图像,处理后打印和展示处理后信息。
支持操作系统:Windows、Ubuntu 支持操作系统:Windows、Ubuntu
建议Python 版本: 3.10, 3.11, 3.12 建议Python 版本: 3.10, 3.11, 3.12
@@ -45,7 +43,7 @@ mkdir orisys_ws
```bash ```bash
cd "C:\Users\orisys\orisys_ws" cd "C:\Users\orisys\orisys_ws"
``` ```
### 安装 CUDA 和 cuDNN(可选) ### 安装 CUDA(可选)
如果您拥有 NVIDIA 显卡,并希望利用 GPU 加速计算,则需要安装 CUDA 工具包和 cuDNN。 如果您拥有 NVIDIA 显卡,并希望利用 GPU 加速计算,则需要安装 CUDA 工具包和 cuDNN。
1. 方法一(官网下载): 1. 方法一(官网下载):
@@ -58,7 +56,7 @@ cd "C:\Users\orisys\orisys_ws"
conda install nvidia/label/cuda-12.9.0::cuda-toolkit nvidia::cudnn conda install nvidia/label/cuda-12.9.0::cuda-toolkit nvidia::cudnn
``` ```
该方法会自动配置好 CUDA 和 cuDNN,无需手动下载。 该方法会自动配置好 CUDA,无需手动下载。
注意:如果您的计算机没有 NVIDIA 显卡,或者不想使用 GPU 加速,可以跳过此步骤。套件也支持纯 CPU 运行(但速度可能较慢)。 注意:如果您的计算机没有 NVIDIA 显卡,或者不想使用 GPU 加速,可以跳过此步骤。套件也支持纯 CPU 运行(但速度可能较慢)。
@@ -84,9 +82,9 @@ pip install --upgrade orisys
### 使用 .whl 文件安装(离线或指定版本) ### 使用 .whl 文件安装(离线或指定版本)
从官方渠道获取 .whl 文件,文件名格式通常为: 从官方渠道获取 .whl 文件,文件名格式通常为:
orisys-{版本}-{Python版本}-{Python版本}-{操作系统}_{硬件架构}.whl orisys-{版本}-{Python版本}-{Python版本}-{操作系统}_{硬件架构}.whl
例如:orisys-0.5.0-cp310-cp310-win_amd64.whl 表示: 例如 `orisys-<版本>-cp310-cp310-win_amd64.whl` 表示:
版本:0.5.0 版本:`<版本>`(以实际下载的文件名为准,可用 `python -c "import orisys; print(orisys.__version__)"` 确认)
Python 版本:3.10(cp310) Python 版本:3.10(cp310)
操作系统:Windows(win) 操作系统:Windows(win)
硬件架构:64位(amd64) 硬件架构:64位(amd64)
@@ -97,7 +95,7 @@ Python 版本:3.10(cp310)
```bash ```bash
# 请将文件名替换为您实际下载的文件名 # 请将文件名替换为您实际下载的文件名
pip install orisys-0.5.0-cp310-cp310-win_amd64.whl pip install orisys-<版本>-cp310-cp310-win_amd64.whl
``` ```
### 安装 CuPy(可选) ### 安装 CuPy(可选)
@@ -139,88 +137,66 @@ python
import orisys import orisys
print(orisys.__version__) print(orisys.__version__)
``` ```
如果输出版本号(例如 0.5.0),则说明安装成功。 如果打印出版本号,则说明安装成功。
## 使用方法 ## 使用方法
## 使用设备专属的力标定参数
摄像头编号与设备 ID 是两个独立参数。命令中的 `-v 0` 或 `--camera_id 0` 只表示选择第 0 个视频源;`device_id` 才是传感器的设备编号(SN),目前需要人工确认并填写。
将**这台设备实际测得**的标定 JSON 放在自行选择的目录中,不要放进 SDK 的 `config/`,也不要覆盖 `sensors/<产品>/force_calibration.json`。目录结构如下,路径可由用户更改:
```te
D:/orisys_data/calibrations/
├── box/
│ └── BOX-001.json
├── mini/
│ └── MINI-001.json
└── finger/
└── FINGER-001.json
```
例如,使用 `BOX-001` 时,`calibration_dir` 指向上图的 `calibrations` 目录,`device_id` 填 `BOX-001`:
```python
import orisys
sensor = orisys.Sensor(
0, # 摄像头编号,不是设备 ID
config_name="box",
calibration_dir="D:/orisys_data/calibrations",
device_id="BOX-001",
verbose=True,
)
try:
print(sensor.calibration.source_path if sensor.calibration else "未加载标定")
print(sensor.force_units) # 依次为 FN、FX、FY 的单位
finally:
sensor.disconnect()
```
或者使用命令:
```bash
python examples/basic_pipeline.py -v 0 -c box --calibration-dir data/calibrations --device-id BOX-001
```
若该设备的标定文件不存在,SDK 不应用标定,力值保持 raw,也不会回退到产品默认标定。若文件存在,但其中的设备 ID、产品类型或测量设置与当前设备不匹配,SDK 会报错。FingerRuntime.open() 支持这两个参数,但 finger_pipeline.py 目前还没有对应的命令行参数。
### 运行示例程序 ### 运行示例程序
套件里面包含了多个例子。运行示例前,请先进入 SDK 示例所在的文件夹,建议在仓库根目录运行(即 `config/`、`examples/`、`assets/` 同级目录)。如果您是通过 PyPI 安装并且本地没有示例文件,可以从上方项目链接下载示例。按产品入口如下:
1. `examples/basic_pipeline.py`:ORY-BOX 复眼/拼接传感器基础 OpenCV 示例(默认配置 `box`) 示例的完整清单、每个产品对应的入口、全部命令与参数,见 [`examples/README.md`](../examples/README.md)。这里只给第一次运行的最短路径。
2. `examples/mini_pipeline.py`:ORY-MINI 基础 OpenCV 示例(默认配置 `mini`)
3. `examples/finger_pipeline.py`:ORY-FINGER 2D 显示最小流程(默认配置 `finger`)
4. `examples/finger_pipeline_3dview.py`:ORY-FINGER Qt 3D 可视化界面(默认配置 `finger`)
`finger` 是当前 ORY-FINGER 的产品配置名。 先在仓库根目录(即 `config/`、`examples/`、`assets/` 同级)确认传感器 ID:
下面先介绍最基础的 ORY-BOX 运行方式。 - Windows:在 PowerShell 运行 `Get-PnpDevice -PresentOnly | Where-Object { $_.Class -match '^Camera' }`。若插上传感器后多出一行,ID 为 `1`,否则为 `0`。
- Ubuntu:运行 `v4l2 --list-devices`,从新增的设备名取编号(`/dev/video0` 即 ID `0`)。
#### Windows 然后运行(假设 ID 为 `0`):
1. 检查是否有其他正在连接的 USB 摄像头设备:打开 PowerShell,运行命令 `Get-PnpDevice -PresentOnly | Where-Object { $_.Class -match '^Camera' }`,如果没有的话不会有输出。
2. 把传感器的 USB 头与电脑连接,再次运行步骤一的命令,会新增一行输出。
3. 如果第一步没有输出,传感器的 ID 就是 `0`;如果第一步有一个设备,传感器的 ID 就是 `1`,如此类推。
4. 在确定了传感器的 ID(下方例子假设为 `0`)后,打开 Anaconda Prompt,切换到工作路径,然后在命令行中输入:
```bash
python examples/basic_pipeline.py -v 0 -c box
```
5. 程序会运行,并自动打开可视化窗口和打印资料。在窗口上按键盘 `q` 键可以停止运行,按 `r` 键可以重设数据。
#### Ubuntu
1. 检查是否有其他正在连接的 USB 摄像头设备:打开终端,运行命令 `v4l2 --list-devices`,如果没有的话不会有输出。
2. 把传感器的 USB 头与电脑连接,再次运行步骤一的命令,会新增输出。
3. 根据新增的输出,例如 `/dev/video0`,确定传感器的 ID 为 `0`;如果是 `/dev/video3`,ID 为 `3`,如此类推。
4. 在确定了传感器的 ID(下方例子假设为 `0`)后,打开终端,激活相应的 Python 环境,切换到工作路径,然后在命令行中输入:
```bash
python examples/basic_pipeline.py -v 0 -c box
```
5. 程序会运行,并自动打开可视化窗口和打印资料。在窗口上按键盘 `q` 键可以停止运行,按 `r` 键可以重设数据。
#### ORY-FINGER 示例
如果您使用的是 ORY-FINGER(配置名 `finger`),可运行:
```bash ```bash
python examples/finger_pipeline.py --camera_id 0 --config finger python examples/basic_pipeline.py -v 0 -c box
``` ```
如果输入来自视频文件,也可以改用: 窗口打开后按 `q` 停止,按 `r` 重设数据。若本地没有示例文件(例如通过 PyPI 安装),从项目链接下载。
```bash
python examples/finger_pipeline.py --video ./data/sample.mp4 --config finger
```
该示例会:
- 读取相机或视频源
- 将图像展开到平面
- 计算光流、法向/切向力和接触区域
- 在 OpenCV 窗口中显示展开后的光流箭头图
常用参数:
- `--verbose`:输出更详细的初始化日志
例如:
```bash
python examples/finger_pipeline.py --camera_id 0 --config finger
```
#### ORY-FINGER 3D 可视化界面
如果要使用 GUI,先安装 GUI 依赖:
```bash
pip install "orisys[gui]"
```
然后运行:
```bash
python examples/finger_pipeline_3dview.py --video 0 --config finger
```
该界面会提供:
- 手指平面化光流箭头显示
- 力曲线
- 3D 视图
### 开发套件使用方法 ### 开发套件使用方法
参考开发套件附带的开发文档。 参考开发套件附带的开发文档。
+139
View File
@@ -0,0 +1,139 @@
---
kind: reference
---
# Plan: move the GUI out of the SDK into `orisys-studio`
Status: **planned, then postponed (2026-09-30).** Nothing here is in progress. This document
records what would move, what would stay, and the one coupling that makes the move non-trivial, so
that the work can be picked up without re-deriving it. It is a plan, not a description of current
behaviour — everything described below is still in the SDK exactly as listed.
## Why
`examples/gui/` is not an example. It is a full Qt + open3d + pyqtgraph application — 30 modules,
4,815 lines — that ships inside the SDK wheel. The GUI belongs to the `orisys-studio` application,
which already exists for that purpose. Leaving it in the SDK means the SDK ships, versions and
installs an application it is not responsible for.
## Current state
| Path | Modules | Lines | What it is |
|---|---|---|---|
| `examples/gui/` | 30 | 4,815 | The application: `viewer2d/`, `viewer3d/`, `widgets/`, `app.py`, `log_config.py` |
| `examples/viz/` | 11 | 2,298 | **Shared** drawing/rendering helpers — see the coupling below |
| `examples/pipeline/` | 6 | 577 | Sensor service, CSV recorder, profiling. **SDK support code, not GUI** |
### Good news: the SDK core does not depend on the GUI
Verified by AST, not by grep: **no module under `src/orisys/` imports `gui`, `viz`, `pipeline` or
`examples`.** The only match is `src/orisys/__init__.py` importing `orisys.pipeline`, which is the
SDK's own module. The dependency runs one way, so the GUI can leave without touching the library.
Two apparent hits are prose, not code: `src/orisys/configs.py` mentions `examples/` in a docstring
about run-root layout, and `export_sensors.py` has an `Examples::` docstring section. The
`configs.py` anchor is a real behavioural dependency to preserve — the run-root heuristic keys off
a sibling `examples/` directory, so that directory must keep existing in the packaged layout even
after `gui/` leaves it.
### The coupling: `viz/` is shared, so it cannot simply follow the GUI
```
examples/gui/ ──uses──▶ examples/viz/ ──uses──▶ orisys (SDK)
examples/pipeline/service.py ──uses──▶ examples/viz/
tests/test_contact_arrow_scale.py ──uses──▶ examples/viz/
```
`examples/pipeline/service.py` and a test depend on `viz`, and `viz` is a mix: `mesh_camera.py`,
`contact_arrow.py`, `flow_arrow.py` and `offscreen_mesh.py` are open3d rendering helpers, while
`render_2d.py`, `recording.py`, `signal_filters.py` and `force_charts.py` are toolkit-independent.
**Moving `viz/` wholesale would break `examples/pipeline/`.** It has to be split, not moved.
### The real coupling is in the tests, and it is inverted
`tests/test_qt_viewer_backend.py` imports `examples/gui` directly, and `tests/conftest.py` inserts
`EXAMPLES_ROOT` into `sys.path` with a guard whose stated reason is that **`examples/gui` must win
over `build/gui`, because both are packages named `gui`**. So the SDK's test suite currently
depends on the GUI application being present and on a name-resolution guard existing for it.
This is the inversion to fix: the SDK must not depend on the GUI.
Two related facts:
- `examples/gui/` contains committed `__pycache__` directories (`adapter.cpython-310.pyc`,
`cpython-313.pyc`, …) — build artefacts tracked alongside source.
- `tests/test_wheel_contents.py` and `tests/test_pyproject.py` assert on what the wheel bundles, so
they will need updating when the `gui/**/*.py` bundle pattern goes.
## What moves, what stays
**Moves to `orisys-studio`:**
- `examples/gui/**` — the whole application, as the single home for GUI code.
- `examples/finger_pipeline_3dview.py` — a 91-line launcher for the dashboard; its only purpose is
to start the GUI, so it belongs with the GUI.
- The open3d rendering half of `examples/viz/`: `contact_arrow.py`, `flow_arrow.py`,
`mesh_camera.py`, `offscreen_mesh.py`, plus the open3d-dependent `__init__` re-exports.
- `tests/test_qt_viewer_backend.py` — it tests the GUI, so it moves with the GUI.
**Stays in the SDK:**
- `examples/pipeline/` — SDK support code (sensor service, CSV recording, profiling).
- `examples/viz/` toolkit-independent half: `render_2d.py`, `recording.py`, `signal_filters.py`,
`force_charts.py` — needed by `examples/pipeline/service.py`.
- `examples/plot_data.py` — uses pyqtgraph for force charts only; 96 lines. Decide separately
whether it follows the GUI; it is the one remaining example with a Qt dependency after the move.
- `src/orisys/finger/viewer.py` (`UVMeshViewer`) — **see the deprecation note below.**
## The public-API problem, handled separately
`UVMeshViewer`, `init_uv_mesh_viewer` and `update_uv_mesh_viewer` are exported in
`src/orisys/finger/__init__.py`'s `__all__`. **Removing them is a breaking change to a released
API**, so it must not be bundled into a code move:
1. Release with them present and documented `DeprecationWarning`s emitted on construction.
2. Remove them at the next major version, with the release notes naming the replacement.
The audit (`docs/api-audit.md`) already lists `UVMeshViewer` among the surface that is used by
examples and absent from the reference.
## Sequence, lowest risk first
| # | Step | Risk | Notes |
|---|---|---|---|
| 1 | Add `docs/plan-gui-extraction.md` (this file) | none | Done |
| 2 | Delete the committed `__pycache__` under `examples/gui/` and add it to `.gitignore` | none | Pure hygiene; ship independently |
| 3 | Tag the SDK at the commit whose `examples/gui` is known-good, so the studio move has a pinned source | none | A tag, not a submodule or subtree |
| 4 | In `orisys-studio`: import `examples/gui` + the open3d half of `viz` as its own provenance-stated change, reconciling against its existing viewer code | **medium** | The studio repo already has viewer code; this is a merge decision, not a copy |
| 5 | In the SDK: move `tests/test_qt_viewer_backend.py` out, and simplify the `conftest.py` guard | **medium** | The guard exists for `gui`; with `gui` gone, `EXAMPLES_ROOT` may still be needed for `viz`/`pipeline`. Re-derive, do not guess |
| 6 | In the SDK: retire `examples/gui/`, `finger_pipeline_3dview.py`, and the moved `viz` modules | low once 4 and 5 land | |
| 7 | Update the bundle patterns in `pyproject.toml` (drop `gui/**/*.py`, adjust `viz/**`) and the wheel tests | low | `test_wheel_contents.py`, `test_pyproject.py` |
| 8 | Update `examples/README.md` and the SDK reference to stop pointing at the dashboard | low | The README already says the GUI is moving |
| 9 | Deprecate `UVMeshViewer` in a release; remove at the next major | low | Separate release cycle |
Steps 2 and 3 are safe to do immediately and independently. Steps 4–5 are the actual move and need
both repositories. Nothing here requires a library behaviour change, which is what makes the move
tractable.
## Open questions for the studio side
- **Merge or adopt?** `orisys-studio` already has its own viewer. If its code is the descendant of
this one, take the SDK copy only where it is ahead; otherwise choose one and delete the other.
This must be decided before step 4, because the answer changes what gets imported.
- **Do the SDK's 2D and 3D viewers need to keep working headless for CI?** `examples/viz` contains
offscreen rendering used to produce evidence images. If any SDK or sensor-design workflow depends
on that, the offscreen half may need to stay reachable from the SDK even after the app moves.
- **Does `plot_data.py` follow the GUI?** It is an example, not an application, and it is the last
Qt-dependent file in `examples/`.
## Acceptance criteria
- [ ] `src/orisys/` still imports no `gui`/`viz`/`examples` module (re-run the AST check).
- [ ] The SDK test suite passes with `examples/gui/` absent.
- [ ] No module named `gui` ships in the wheel; `test_wheel_contents.py` asserts it.
- [ ] `examples/pipeline/service.py` and `tests/test_contact_arrow_scale.py` still import their
`viz` dependencies.
- [ ] `UVMeshViewer` is either deprecated-but-present or removed by an explicit major-version
decision — never dropped silently in a move.
- [ ] The `examples/` sibling directory still exists in the packaged layout, because
`configs.py`'s run-root anchor depends on it.
+120
View File
@@ -0,0 +1,120 @@
---
title: Build and publish the Orisys SDK
description: Contributor release checklist and the verified build and publish entry points.
---
# Build and publish the Orisys SDK
This guide gives contributors the path from a source change to release evidence.
The build launcher is [build/run.py](../build/run.py), documented in
[build/README.md](../build/README.md). Publishing to GitHub or PyPI changes external
systems and is a maintainer action. The existing
[Chinese release runbook](publishing.md) retains historical details; verify its
version-specific examples against current code before using them.
## Before building
Run the non-hardware tests from the repository root with the intended Python
environment:
```bash
python -m pytest -m "not slow and not hardware"
```
Check the current SDK version at `src/orisys/__init__.py`. `pyproject.toml` reads it
as a dynamic version. Do not copy a previous release number into new commands or
documentation. Check the source diff and confirm the intended product bundles and
assets are present; `pyproject.toml` declares which non-code files are packaged.
## Build the candidate
Use the launcher appropriate to the target. `wheel-win` invokes the Windows build;
`wheel-manylinux` invokes the Docker-backed Linux build. A single-interpreter `wheel`
build is useful for a local candidate, but does not cover every supported ABI.
```bash
python build/run.py wheel-win
python build/run.py wheel-manylinux
python build/run.py wheel-manylinux -Architecture aarch64
```
Windows builds use the configured `py310`, `py311`, and `py312` environments.
The Linux build needs its documented Docker setup. Built wheels are placed under
`SDK/<version>/`; Linux upload candidates may also appear in `dist/repaired/`.
See [build/README.md](../build/README.md) for build prerequisites and narrower
commands. A contributor without these environments can still run tests and hand
off a source change; do not describe it as a verified release candidate.
Build Windows x64, Linux x86_64 and Linux ARM64 before preparing the release
report. With Python 3.10, 3.11 and 3.12 this produces nine wheels. Both publishers
use every wheel in `SDK/<version>/`; files added or rebuilt after preflight
invalidate its certification. ARM64 CPU wheels do not require CUDA.
## Run the release gate
After building, run `release preflight` on a clean, reviewed commit. It checks
repository state, version consistency, tests, bundled assets, wheel contents, and
an isolated runtime on recorded frames. `--check-pypi` adds a read-only duplicate
version check against PyPI. The report is written under
`build/release/reports/<version>/`.
```bash
python build/run.py release preflight --all-wheel-pythons --check-pypi
```
`--all-wheel-pythons` checks every staged ABI and architecture. Run the complete
mixed-platform gate from Windows: Windows wheels use the `py310`, `py311` and
`py312` conda environments, while Linux wheels use `orisys-manylinux` or
`orisys-manylinux-aarch64` with the matching container Python. Docker must already
have these locally built images; verification never pulls or rebuilds them.
Linux dependencies come from each image's ABI wheelhouse, in an isolated venv.
ARM64 verification on Intel/AMD uses QEMU and can take longer. Every wheel must
pass imports, bundled-asset loading and the CPU video pipeline; emulation does
not certify Q6A or Jetson hardware, GPU or NPU behavior.
For work in progress, `--allow-dirty --keep-going` can collect diagnostics, but
that report does **not** certify a release. Do not use `--skip-tests`, `--no-runtime`,
or `--no-assets` to turn missing verification into approval. If the runtime clip is
missing, provide `--video <path>` or `ORISYS_VERIFY_VIDEO`; the gate should report
failure rather than silently claiming a pass.
For one existing wheel, `release audit <wheel>` checks it without preparing a
complete release. Add `--assets --runtime` when a suitable verification clip and
environment are available. This is useful for investigating a candidate; it does
not replace the full preflight gate.
## Release handoff and publication
Hand the maintainer the version, source commit, wheel filenames, preflight report,
and any hardware or platform checks that were not run. The publish commands read
the passing preflight report and verify the wheel hashes again.
The report must also contain successful runtime evidence matching every wheel's
hash. Older reports that checked only Windows wheels must be regenerated.
```bash
python build/run.py release github --version <version>
python build/run.py release pypi --version <version>
```
These commands omit `--apply`, so they preview the publish plan. The GitHub preview
may still fetch remote tags; the PyPI preview may check credentials and the index.
Maintainers run the commands with `--apply` only after checking the target repository,
release notes, credentials, and candidate files. `release pypi --apply` targets
TestPyPI; publishing to the production index also requires `--confirm-pypi`.
GitHub publication can sync the release mirror, commit, tag, push, and create a
Release. See the launcher help and [build/README.md](../build/README.md) for the
exact flags. A published version cannot be overwritten, so do not treat a rerun as
a way to replace bad wheels.
The release order is build all candidates, commit source changes, run the clean
preflight, preview both publishers, then execute the authorized publications.
GitHub attaches all certified wheels and generates notes naming their actual
platforms, including ARM64. PyPI defaults to TestPyPI first; after confirming it,
use `--confirm-pypi --apply` for production. Each upload checks the complete
filename set and downloads every Python/platform wheel from that index to
compare its hash. TestPyPI download verification does not fall back to PyPI.
If downloads are not yet available, the command reports the successful upload
as unconfirmed and stops before uploading to later indexes; check propagation
before rerunning.
If a published version has a different file set, the commands stop rather than
silently extending or replacing that release.
+399
View File
@@ -0,0 +1,399 @@
# 发布流程(构建 → 校验 → 发布)
本文档说明如何构建 orisys 的 Nuitka 二进制 wheel、如何在发布前用一条命令做完整门禁,以及当前仍需人工执行的发布步骤。
> 历史说明:`README.md` 与 `build/wheel/docker/run.ps1` 早就引用了本文件,但它此前并不存在。本文件补齐该引用。
---
## 0. 环境准备
推荐用 `py310` conda 环境执行构建与校验(它同时具备 pytest / twine / numba / Nuitka / open3d / PyQt5):
```powershell
conda activate py310
python -m pytest -m "not slow and not hardware" # 开发期快速回归
```
`py311`、`py312` 仅作为 Nuitka 各 ABI 的编译环境按需调用(`build/wheel/win.bat` 会依次使用 `py310`/`py311`/`py312`)。
**不要**在同一个环境里既装旧版 `orisys` 又从源码目录跑测试:site-packages 里的旧包会静默遮蔽 `src/orisys`。
`tests/test_harness_guards.py` 会显式报错提示执行 `pip uninstall -y orisys`。
---
## 1. 构建 wheel
```powershell
# Windows:py310 / py311 / py312 三个 ABI
python build/run.py wheel-win
# 单个解释器(当前环境)
python build/run.py wheel
# Linux manylinux(Docker Desktop,Linux 容器模式)
python build/run.py manylinux
python build/run.py wheel-manylinux -PythonVersions 3.10
```
产物落点:
- `SDK/<version>/` — 交付用 wheel(git 忽略)
- `dist/repaired/` — 上传候选(git 忽略)
---
## 2. 发布前门禁:`release preflight`
```powershell
# --version 可省略:默认取 pyproject 的 dynamic version(即 src/orisys/__init__.py 里的 __version__)
python build/run.py release preflight
```
按顺序执行 8 个步骤,任一步失败即整体失败(`--keep-going` 可改为收集全部失败):
| 步骤 | 检查内容 |
| --- | --- |
| `repo-state` | 是 git 仓库、工作树干净、记录 HEAD(`--allow-dirty` 可豁免,用于提交前预检) |
| `version-consistency` | `src/orisys/__init__.py::__version__` == pyproject 的 dynamic attr == `SDK/<version>/` 中 wheel 文件名里的版本 |
| `version-strings` | 追踪的文本文件里没有与当前版本不符的版本字面量(见 §5.3) |
| `duplicate-guard` | 该版本是否**已经发布过**:本地 git tag(`v<version>` / `<version>`,含 sibling `orisys-sdk-release` 仓库)命中即拒绝;`--check-pypi` 时再查 PyPI。`SDK/<version>/` 里已有 wheel 只作信息展示——那是 preflight 的正常输入,不是错误 |
| `tests` | `pytest -m "not slow and not hardware"`(`--longrun` 时包含 slow) |
| `bundle-precheck` | 每个 `[[tool.orisys.bundle.items]]` 至少解析出 1 个文件;finger 必需资产存在 |
| `wheel-inspection` | 逐个 wheel 做静态断言:文件名/版本/METADATA 字段/RECORD 哈希/压缩包路径/体积上限/无 pyc 与缓存/无 `orisys/**/*.py` 源码泄漏/与源码树逐文件字节一致 |
| `wheel-runtime` | 每个 wheel **一个** venv,在里面依次做:导入冒烟 → 内置 configs/sensors 资产可加载 → 对真实视频跑 40 帧公开流水线(无需相机、无 GUI) |
正常顺序是**先构建、再 preflight**,不需要额外开关:
```powershell
python build/run.py wheel-win
python build/run.py manylinux
python build/run.py release preflight --all-wheel-pythons --check-pypi
```
提交前想先看结果(此时工作树通常还是脏的):加 `--allow-dirty --keep-going`。
产物报告:
- `build/release/reports/<version>/preflight.json` — 机器可读,含每个 wheel 的 sha256,供后续发布步骤复用
- `build/release/reports/<version>/preflight.txt` — 人工阅读
- `build/release/reports/<version>/junit.xml` — pytest 结果
(`build/release/reports/` 已加入 `.gitignore`。)
### 常用参数
| 参数 | 作用 |
| --- | --- |
| `--wheel-dir DIR` | 校验指定目录下的 wheel(默认 `SDK/<version>`) |
| `--all-wheel-pythons` | cp310/cp311/cp312 各自在对应 conda 环境里独立校验 |
| `--video PATH` / `--frames N` / `--config NAME` | 覆盖运行时校验输入(默认 `data/sample/s1_480p.mp4` / 40 帧 / `box`) |
| `--offline` | 复用父环境的 numpy/opencv/numba,不去 PyPI 装依赖 |
| `--check-pypi` | 唯一一处网络调用:只读 PyPI,防止重复发布 |
| `--skip-tests` / `--no-runtime` / `--no-assets` | 显式跳过并在报告中留痕;**跳过的运行不能作为发布依据** |
| `--report-dir DIR` | 覆盖报告根目录 |
`wheel-runtime` **不会静默跳过**:找不到视频时直接失败,提示 `--video` / `ORISYS_VERIFY_VIDEO`。
### 单体体检:`release audit`
对单个 wheel 做静态断言(+ 可选运行时校验),不需要版本上下文。适合体检历史产物:
```powershell
python build/run.py release audit SDK\<version>\orisys-<version>-cp312-cp312-win_amd64.whl
python build/run.py release audit dist\repaired\orisys-<version>-cp310-cp310-win_amd64.whl --strict-bundle
# 追加运行时校验(--assets 与 --runtime 同时给时共用一个 venv,只装一次依赖)
python build/run.py release audit <wheel> --assets --runtime --video data/sample/s1_480p.mp4
```
默认对「内置文件与当前源码树不一致」只告警(历史 wheel 与当前分支不一致是正常的);加 `--strict-bundle` 改成失败(刚构建完的 wheel 应当一致)。
旧的导入冒烟测试仍然可用,行为不变:
```powershell
python build/run.py verify <wheel>
python build/run.py verify <wheel> --contents # 静态断言
python build/run.py verify <wheel> --assets --runtime # 资产 + 真实帧
python build/run.py verify --gui-dist --product ory_finger
```
---
## 3. 发布(两条命令)
先决条件:`release preflight` PASS,**而且是在干净提交上跑的**。两个发布命令都只发布「preflight 认证过的东西」——它们会读取 `build/release/reports/<version>/preflight.json`,逐个核对 wheel 的 sha256,并拒绝使用带豁免项(`--skip-tests` / `--allow-dirty` / `--no-runtime` / `--force`)的报告。所以正确顺序是:
```powershell
git add -A; git commit -m "..."
python build/run.py wheel-win && python build/run.py manylinux
python build/run.py release preflight --all-wheel-pythons --check-pypi # 干净树 + 无豁免
```
两个命令**默认都是 dry run**,只打印将要执行的动作;必须显式加 `--apply` 才会真正动手。GitHub 的 `gh release create`、`git push`,以及 PyPI 上传(版本号永久占用、不可覆盖)都属于不可撤销操作。
### 3.1 GitHub:同步公开仓库 + 上传 Release
```powershell
python build/run.py release github --version 0.5.2 # 预览
python build/run.py release github --version 0.5.2 --apply # 执行
```
执行顺序:
1. 认证书校验(上面那份报告 + 逐 wheel sha256)
2. `git fetch --tags`,若 `v0.5.2` 或 `0.5.2` 已存在则拒绝;`gh release view` 已存在也拒绝
3. 按 `tools/downstream_sync_manifest.json` 同步公开内容到 `../orisys-sdk-release`,并运行 manifest 里的 `checks`
4. 把 `SDK/0.5.2/*.whl` 放到 `orisys-sdk-release/release/0.5.2/`(该目录在 release 仓库里是 gitignore 的,只用于上传附件)
5. 提交 → 打 tag `v0.5.2` → push 分支与 tag
6. `gh release create v0.5.2 --verify-tag --latest <6 个 wheel>`,然后回读 `gh release view --json assets` 核对附件齐全
需要:`gh` 已安装并登录(`winget install --id GitHub.cli` 然后 `gh auth login`)。没装 gh 时:
- dry run 仍会完整打印计划(只告警)
- `--apply` 会失败,或改用 `--skip-release-upload`:同步、提交、打 tag、push 都自动完成,然后打印出你自己要跑的 `gh release create`
其他开关:`--notes-file`(自定义 release notes)、`--attach-preflight`(把 `preflight.json` 也作为附件上传,便于审计;注意报告里有本机绝对路径)、`--allow-dirty-release`、`--skip-fetch`、`--skip-checks`。
> tag 拼写沿用历史风格 `v<version>`(注意历史里那个 typo tag `0.50` 指向 `v0.5.1`)。命令会在动手前拒绝已存在的 tag,所以不会重复发布。**如果 tag 已经 push 但 Release 上传失败(例如 `gh` 没配好),直接重跑同一条 `--apply` 命令即可续跑**:它会跳过创建 tag,只补做剩下的步骤;只有「GitHub Release 已存在」才拒绝。
### 3.2 PyPI:TestPyPI 先行,确认后再上正式仓库
```powershell
python build/run.py release pypi --version 0.5.2 # 预览
python build/run.py release pypi --version 0.5.2 --apply # 只发 TestPyPI
python build/run.py release pypi --version 0.5.2 --confirm-pypi --apply # TestPyPI + 正式 PyPI
# 只有正式 token、没有 TestPyPI 账号时:跳过彩排直接发正式库
python build/run.py release pypi --version 0.5.2 --pypi-only --apply
```
`--pypi-only` 与 `--confirm-pypi` 互斥。**注意它的代价**:没有 TestPyPI 彩排,而正式库的版本号一旦占用就无法复用、无法替换。它仍然会做完整门禁、`twine check`、上传前查重、上传后回读索引核对文件、以及把产物重新拉回来验证。
执行顺序:
1. 认证书校验(与 `release github` 同一份报告)
2. `twine check` 全部 wheel(校验 METADATA/long description,上传前就能发现问题)
3. 只读查询目标索引,确认该版本不存在(TestPyPI 与 PyPI 各查各的)
4. 上传 → **轮询**索引确认 6 个文件都在 → `pip download` 把刚上传的 wheel 拉回来,证明产物可获取
5. 未加 `--confirm-pypi` 时只发 TestPyPI,然后停在这里并打印后续命令
两个针对「索引不即时一致」的设计(TestPyPI 尤其明显,上传成功后 JSON API 可能滞后若干秒):
- **上传后的确认会重试**(默认 6 次 × 5 秒)。确认不了也**不算失败**:twine 上传成功就说明产物已发布,命令会打印警告、给出项目页链接并以 0 退出,不会把一个成功的上传报成失败。真正的失败只有两种:twine 上传本身失败,或索引上的文件集与本地不一致。
- **重复运行是安全的**:若索引上已经存在该版本且文件集完全一致,该阶段会被视为「已完成」而跳过(`[ok] testpypi already serves orisys 0.5.2 with all 6 file(s)`),不会重传、也不报错。这让你在确认不明确时可以直接重跑。如果文件集**不一致**(索引上是另一次构建),才会失败并停下。
所以「上传后确认失败」不等于要重新上传——索引版本号不可覆盖,重跑只会做一次只读核对。
### 3.3 凭证:`release auth` 与 `~/.pypirc`
```powershell
python build/run.py release auth # 查看两个索引的凭证状态
python build/run.py release auth --set-token pypi # 隐藏输入,写入 ~/.pypirc
python build/run.py release auth --set-token testpypi
python build/run.py release auth --repair # 修「有段但没列进 index-servers」
```
`--set-token` 的 token 只从**隐藏输入**或 `--token-stdin`(供脚本/CI)读取,**不接受命令行参数**——避免落进 shell 历史和进程列表。写入采用定点文本编辑,文件里的说明注释会保留;同时会把该索引补进 `[distutils] index-servers`(这一步很关键,见下)。
`release auth` 只有在你真的运行 `--set-token` 时才写文件;不带参数时只读并报告,状态判断比「文件是否存在」严格:
```
config file: C:\Users\cheun\.pypirc (exists)
[-- ] [pypi] not usable: [pypi] still contains a placeholder token
[-- ] [testpypi] not usable: C:\Users\cheun\.pypirc has no [testpypi] section
```
**占位符、空密码、`username` 不是 `__token__`,都会被判定为「不可用」**——否则 twine 只会在真正上传时才发现,而那时已经动过索引了。
`release pypi` 在 `--apply` 时若凭证缺失:有终端就地提示输入(隐藏),无终端则直接失败并打印 `release auth --set-token <index>`;也可显式加 `--prompt-credentials`。dry run 只告警并继续展示计划,不会卡在输入上。
#### twine 直接读取的格式
默认路径 `~/.pypirc`(Windows 上即 `C:\Users\<你>\.pypirc`,由 `USERPROFILE` 展开)。**没有**环境变量可以改这个路径;要用别的文件就传 `--config-file`。
```ini
[distutils]
index-servers =
pypi
testpypi
[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-你的正式token
[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-你的TestPyPI token
```
容易搞错的点:
- **`[distutils] index-servers` 必须列出该索引,否则 twine 完全忽略那一节** —— 这是最常见的坑:文件里明明有 `[testpypi]` 段,twine 却报 `InvalidConfiguration: Missing 'testpypi' section`。用 `release auth --repair` 修(把已存在但未列出的段补进 `index-servers`)。
- 更细的语义差别:`index-servers =` **值为空**表示「一个都不认」;只有**整行缺失**时 twine 才会回退到默认的 `pypi testpypi`。所以手工编辑时别把名字删空。
- `username` 就是字面量 `__token__`,不是账号名或邮箱
- 两类 token **都以 `pypi-` 开头**,但**不能互换**:拿正式 token 打 `test.pypi.org` 会被 403 拒绝
- TestPyPI 是**独立服务、独立账号**,注册在 <https://test.pypi.org/account/register/>,你的 PyPI 账号在那里不存在
- 正式 token 建议限定到 `orisys` 项目(项目已存在,可以建 project-scoped token)
- 该文件必须在**任何 git 仓库之外**(放在用户目录里);token 一旦提交进仓库,即使后来删文件也还在历史里——那属于安全事故
- 临时替代方案:会话内设 `TWINE_USERNAME=__token__` + `TWINE_PASSWORD=<token>`,命令会识别这种情形
- PyPI token **没有只读的校验方式**,有效性只能在真正上传时才会被索引确认;所以所有本地检查、查重都排在写操作之前
`release auth` 的「可用」判定**直接问 twine 自己**(`twine.utils.get_config` 解析出的 username/password),而不是自己解析 ini,这样报告结果与 twine 的行为不会分叉。twine 未安装时才回退到本地复刻同一套规则。
只需要 TestPyPI 时才把 `testpypi` 写进 `index-servers` 并填对应 token;只发正式库时 `--pypi-only` 用不到 `[testpypi]` 段。
### 3.4 已知的历史发布空洞
| 版本 | 本地 `SDK/` | PyPI | GitHub Release |
| --- | --- | --- | --- |
| 0.4.2 | — | 有 | 有 |
| 0.4.3 | — | 有 | 有 |
| 0.4.4 | 有 | **无** | **无** |
| 0.5.0 | 有 | **无** | **无** |
| 0.5.1 | 有 | 有 | 有(tag `v0.5.1` + typo tag `0.50`) |
| 0.5.2 | 有 | **待发布** | **待发布**(`release preflight --check-pypi` 确认 PyPI 尚无 0.5.2) |
> 本表按 2026 年 0.5.2 构建时的实际状态填写。`0.5.1` 这一行曾被一次全局替换误改成 `0.5.2` 并丢失,请勿再用「全局把 0.5.1 替换成新版本号」的方式改本文件——历史事实不应该被改。
---
## 4. 环境变量
| 变量 | 用途 |
| --- | --- |
| `ORISYS_TEST_VIDEO` | pytest 用的视频;默认自动取 `data/sample/s1_480p.mp4` |
| `ORISYS_TEST_CONFIG` | pytest 用的配置名;默认自动挑一个含 `stitcher.box_size` 的配置 |
| `ORISYS_VERIFY_VIDEO` | 运行时校验用的视频 |
| `ORISYS_VERIFY_CONFIG` | 运行时校验用的配置名(默认 `box`) |
| `ORISYS_LONGRUN_BACKEND` / `ORISYS_LONGRUN_FRAMES` | `slow` 长稳测试的参数 |
| `CONDA_ENVS_ROOT` | conda envs 根目录(`--all-wheel-pythons` 用它解析 `py310`/`py311`/`py312`) |
---
## 5. 三个必须记住的坑
### 5.1 Nuitka 编译包的 `__file__` 是假的
Nuitka 把整个 `orisys` 编译成单个扩展模块,其 `__init__` 会安装 `nuitka_module_loader`。于是:
```python
>>> import orisys
>>> orisys.__file__ # 合成的、磁盘上并不存在的路径
'...\\site-packages\\orisys\\orisys\\__init__.py'
>>> orisys.__path__ # 第二项是 Nuitka 的哨兵值
['...\\site-packages\\orisys\\orisys', '\\not_existing']
>>> orisys.__loader__.__class__.__name__
'nuitka_module_loader'
```
后果:**在 Nuitka wheel 里不能靠 `orisys.__file__` 或 `importlib.resources.files("orisys")` 定位内置数据**——它们会指向不存在的目录,静默返回「文件不存在」。请改用发行包定位:
```python
from importlib.metadata import distribution
root = distribution("orisys").locate_file("orisys") # site-packages/orisys
```
`build/common/verify/wheel.py` 的内置资产校验就是这么做的,并且会显式断言该目录位于校验用 venv 内,避免被父环境里的旧 `orisys` 顶替。
### 5.2 `--offline` 会被父环境的脏安装污染
`--offline` 通过 venv 的 `--system-site-packages` 复用父环境的 numpy/numba 等依赖,这也顺带暴露了父环境里的 `orisys`。若父环境残留了半卸载的 editable 安装(典型症状是 `pip` 提示 `Ignoring invalid distribution -risys`,且 `site-packages/orisys/__init__.py` 是一个指向死路径的 shim),校验就会读到它。
处置:在构建环境里执行 `pip uninstall -y orisys`,或改用不带 `--offline` 的完全隔离 venv(会联网安装依赖)。
### 5.3 版本号只有一个地方,其他都是推导出来的
版本号的唯一来源是 **`src/orisys/__init__.py::__version__`**。以下全部由它推导,不需要手改:
| 产物 | 推导方式 |
| --- | --- |
| pyproject 的 `[project] version` | `[tool.setuptools.dynamic] version = {attr = "orisys.__version__"}` |
| wheel 文件名里的版本 | `build/wheel/build.py` 调 `get_version()` |
| wheel `METADATA` 的 `Version:` | 生成的 `setup.py` 里写入同一个 `version` |
| `SDK/<version>/` 目录名 | 同上 |
| `importlib.metadata.version("orisys")` | 从 wheel METADATA 读出 |
因此**不要**再往文档、示例、测试里写具体版本号:
- 用法示例写 `SDK/<version>/...`、`--version <version>`,不要写具体版本
- 文档里不要写「版本:x.y.z」;需要时让用户用 `python -c "import orisys; print(orisys.__version__)"` 自己看
- 示例脚本的 docstring 不要再写 `SDK version: x.y.z`
- 测试里需要「一个版本」时用明显是假的版本号(如 `1.x.y` 这种不会与真实版本混淆的值),不要用当前版本 —— 否则每次发版都要改测试,而且一次全局替换就会悄悄改坏断言
`release preflight` 的 `version-strings` 步骤强制这条规则:扫描所有被追踪的文本文件,任何与当前版本不符的版本字面量都会导致失败(`tests/` 与生成物除外;依赖约束如 `numpy>=1.24.4` 由正则排除)。
历史记录属于例外,需显式登记在 `build/release/version.py::VERSION_STRING_ALLOWLIST` 里——登记本身就是「该字面量故意不是包版本」的文档。**不要用全局替换修改含历史版本的文件**:本文件的发布记录表就曾被一次全局替换改错(`0.50` typo tag 被挂到错误的版本上,`0.5.1` 那一行被覆盖丢失)。
### 5.4 wheel 里有什么、没有什么
打包内容由 `pyproject.toml` 的 `[[tool.orisys.bundle.items]]` 单点定义:
| 内容 | 是否进 wheel | 说明 |
| --- | --- | --- |
| `config/*.json` | 是 | `orisys/configs/`,运行时配置解析要用 |
| `sensors/**` | 是 | `orisys/sensors/`,含 finger 的 npz/npy 资产,运行时与 GUI 都要用 |
| `examples/**` | 是 | `orisys/examples/`,用户装完就能跑示例 |
| `docs/**` | **否** | 只随 GitHub 仓库与 PyPI 项目页提供。`docs/publishing.md` 是内部流程文档;`docs/pypi_readme.md` 仅在构建时被读取作为 long description |
因为 docs 不进 wheel,**改文档不会让已构建的 wheel 失效**,不需要重新构建。(改 `examples/`、`config/`、`sensors/` 会——它们确实在 wheel 里,`wheel-inspection` 的 `bundle-fidelity` 会报出来。)
如果以后需要离线 API 参考,加一个只包含 `docs/api.md` 的窄条目即可,不要整个 `docs/` 都塞进去。
---
## 6. 缩短 manylinux 构建时间
`python build/run.py manylinux` 走 `build/wheel/docker/run.ps1` → 容器内 `entrypoint.sh`,对 `cp310/cp311/cp312` **串行**各做一次 Nuitka 编译 + 打 wheel + auditwheel repair + 隔离 venv 校验。
### 6.1 镜像里已经烘好的东西(所以每次构建不必再拉)
`Dockerfile.manylinux` 在每个 ABI 下预置了两类内容:
1. **构建工具**:`pip`/`setuptools`/`wheel`/`nuitka`/`ordered-set`/`zstandard`/`auditwheel`。
因此 `entrypoint.sh` 默认**不再**在每个 ABI 上重跑 `pip install --upgrade pip setuptools wheel auditwheel` —— 那是每次构建 3 次多余的 PyPI 往返,还会让容器里的工具版本与镜像不一致。需要用最新工具时显式开启:
```powershell
docker run ... -e ORISYS_UPDATE_BUILD_TOOLS=1 ...
```
2. **每个 ABI 一个 wheelhouse**(`/opt/orisys-wheelhouse/<tag>`),内含 `numpy`/`scipy`/`opencv-python`/`numba` 及其依赖闭包,外加 `pip`/`setuptools`/`wheel`/`auditwheel` 的 wheel。
`--target pypi-linux` 会**强制**校验:为每个 ABI 建一个一次性 venv,再 `pip install` 刚构建的 wheel 和它声明的运行期依赖。镜像故意不把运行期依赖装进解释器(用户是凭 wheel 元数据装的),所以以前每次构建都要现下约 200 MB × 3。`entrypoint.sh` 现在用 `PIP_NO_INDEX=1` + `PIP_FIND_LINKS=<wheelhouse>` 把 pip 指向本地 wheelhouse:**venv 仍然是隔离的**(没有 `--system-site-packages`),依赖解析与安装依然真实发生,只是不走网络。
找不到 wheelhouse 时会打印警告并回退到联网,不会静默失败。镜像因此增大约 400–500 MB(一次性代价,换每次构建省三次下载)。
### 6.2 不要为了省事直接重建镜像
只改了源码(`src/`、`config/`、`sensors/`、`examples/`、`pyproject.toml`)时加 `-SkipImageBuild`:
```powershell
.\build\wheel\docker\run.ps1 -SkipImageBuild
```
Dockerfile 的 `RUN` 层要装 3 份工具链并下载 3 个 wheelhouse,缓存未命中就是数分钟。**改了 `Dockerfile.manylinux` 或需要更新 wheelhouse 时才重建镜像**。
### 6.3 Nuitka 的排除列表必须完整
`build/wheel/compile.py::NUITKA_NOFOLLOW_IMPORTS` 列出所有**不打进 wheel、运行期从用户环境解析**的第三方包。漏一个意味着 Nuitka 会尝试把它跟进来:编译时间暴涨,而且 wheel 内容开始取决于构建机上恰好装了什么 —— `torch`、`torchvision`(`src/orisys/tracker.py` 的 RAFT 追踪器)和 `pyvkfft`(`backend_availability.py`)就是这样漏掉的,之前只是靠「构建机上没装」才没出事。
`tests/test_nuitka_nofollow.py` 会扫描 `src/orisys` 的全部第三方导入,任何不在排除列表里的都会让测试失败。**新增可选依赖时同步更新列表。**
### 6.4 想要并行?先读这一段
`--jobs` 的默认值是**全部 CPU 数**,单次编译的 C 阶段已经吃满所有核心。真正能靠并行省下的是 Nuitka 的单线程前端(分析/优化/代码生成)、scons 启动、`auditwheel` 与 venv 校验 —— 做法是 3 路并行 + 每个 build 限 `--jobs=<核数/3>`。
但**现在不能直接并行**:`build/common/paths.py` 里的工作目录是写死的共享路径
```
build/nuitka/compile
build/nuitka/wheel
build/nuitka/wheel/repaired
```
而 `build/wheel/build.py` 每次开头都 `_reset_dir()`(`rmtree`)这两个目录,`auditwheel repair` 又会再 `rmtree` 一次 `repaired`。仓库是挂载进容器的(`-v ${RepoRoot}:/src`),所以无论「一个容器 3 路并行」还是「3 个容器并行」,只要共享同一个挂载点,三个 build 就会互相清空暂存目录 —— 表现为随机失败或 wheel 内容错乱。
要并行必须先让工作目录带上 ABI 标签(如 `build/nuitka/<tag>/compile`)。输出路径不用动:`SDK/<version>/`、`dist/repaired/` 的文件名本来就按 ABI 区分,下游 `release preflight` 不受影响。另外 3 个 Nuitka 并行很吃内存,先把 Docker Desktop 内存调到 16 GB 以上再试。
+63
View File
@@ -0,0 +1,63 @@
# Orisys SDK
叠动科技视触觉传感器配套的 Python SDK,提供图像采集、拼接、稠密光流、Helmholtz-Hodge 分解、接触与深度估计,以及可运行的示例。
本文件是 PyPI 项目描述(`long_description`)的源文件:构建时由 `docs/publishing.md` 所述流程从**检出目录**读取(`build/wheel/package.py::PYPI_README_RELPATH`),**不会**被打进 wheel。因此不要在这里写内部仓库布局、构建流水线或发布步骤——那些内容属于仓库内的 `README.md` 与 `docs/publishing.md`。
```bash
python -m pip install orisys
```
支持 Python 3.10 / 3.11 / 3.12,Windows x64 与 Ubuntu x64。
## 快速开始
```bash
python -c "import orisys; print(orisys.__version__)"
python -c "import orisys; print(orisys.list_config())"
```
## 安装选项
- `orisys` — CPU / 基础流程
- `orisys[gui]` — Qt / Open3D 可视化示例
- `orisys[cuda12]` / `orisys[cuda13]` — NVIDIA GPU 加速
- `orisys[gui,cuda12]` — GUI + CUDA 12
`orisys[cuda]` 与 `orisys[cuda12]` 等价。
## 配置
配置解析顺序:
1. 显式 JSON 文件路径
2. 内置配置名(`box`、`mini`、`finger` 等)
3. 产品别名(如 `dd02` -> `dd02-ov`)
4. 向后兼容的 JSON `"name"` 匹配
```python
import orisys
sensor = orisys.Sensor(0, config_name="box")
orisys.export_config("box", "./exported_config.json")
```
## 示例
示例随 wheel 一起分发,位于 `orisys/examples/`。也可以从项目仓库获取:
```bash
python examples/basic_pipeline.py -v 0 -c box
python examples/finger_pipeline.py --camera_id 0 --config finger
```
无相机时可以改用视频文件:
```bash
python examples/finger_pipeline.py --video ./sample.mp4 --config finger
```
## 支持与反馈
- 官网:[https://www.orisys-tech.cn](https://www.orisys-tech.cn)
- 问题反馈:GitHub Issues
+161
View File
@@ -0,0 +1,161 @@
# Sensor bundles and the processing contract
A sensor bundle is a folder that describes one processing profile. This document owns the bundle
descriptor, the resolver, the backend registry and the fingerprints. `sensor.json` keeps its
existing meaning: it is the processing configuration, and its `type` field is the image-processing
arrangement (`compound_eye`, `single_camera`, `uv_unroll`) - never a product family.
## Why the descriptor exists
"Which pipeline processes these frames" used to be inferred from a product name. Those are four
different things, and collapsing them is what made Mini work live but be refused in replay:
| Concept | Meaning | Example |
|---|---|---|
| Product family | What the operator calls it | Box, Mini, Finger |
| Processing backend | Which SDK pipeline processes the images | `compound_eye`, `uv_unroll` |
| Configuration profile | Camera, geometry and algorithm settings for one design revision | `orisys.mini.prototype-r1` |
| Device instance | The physical unit | Serial number, its calibration |
Several families can share one backend, one family can have several profiles, and a recorded clip
has no device instance at all - only a profile and, optionally, a compatible calibration.
## Descriptor
`manifest.json` in the bundle root. Every field except the five identifying keys is optional, so a
bundle with no manifest still resolves through the compatibility adapter.
```json
{
"schema_version": 1,
"profile_id": "orisys.mini.prototype-r1",
"display_name": "Mini prototype R1",
"product_family": "mini",
"processing_backend": "compound_eye",
"sensor_json": "sensor.json",
"default_calibration": "force_calibration.json",
"computational_assets": { "unroll_map": "assets/30fps-0703/unroll_map_uv.npz" },
"assets": { "scene": "scene/finger-display-1.0.0.oryscene" }
}
```
- **`computational_assets`** are required to process at all. A missing one fails the profile.
- **`assets`** are visualization only. A missing one is reported and never blocks processing - a
missing 3D scene is not a missing deformation field.
- Paths resolve relative to the bundle root, so moving or renaming the folder changes nothing.
Absolute paths and `..` escapes are rejected at parse time.
- An unsupported `processing_backend` or `schema_version` raises `DescriptorError`. The supported
descriptor schema version is 1.
## Compatibility for existing bundles
`resolve_bundle` reads a new-shape descriptor when `schema_version` is present. Otherwise - no
manifest, or the older `name`/`sensor_json`/`assets_dir`/`viz_pose` shape - the adapter describes
it. That is the only place a folder name is consulted, and only for the built-in set:
| Folder | Product family | Backend (when the config does not decide) |
|---|---|---|
| `box` | box | `compound_eye` |
| `mini` | mini | `compound_eye` |
| `finger` | finger | `uv_unroll` |
| `finger-mems` | finger-mems | `compound_eye` |
For a bundle without a descriptor the backend follows the configuration, using the rule
`Sensor._load_config` already applies: `type` when present, else `compound_eye` when the config has
a `stitcher` block, else `single_camera`. Only `finger` declares `type` today; `box`, `mini` and
`finger-mems` are decided by the stitcher. A folder that is neither built-in nor carries a
descriptor is rejected - never routed to `compound_eye` because box works that way.
## Resolver
```python
from orisys import resolve_bundle, validate_profile, create_processor, CalibrationSelection
profile = resolve_bundle("sensors/mini") # or any bundle folder
profile = resolve_bundle("sensors/mini", calibration_selection=CalibrationSelection.NONE)
session = validate_profile(profile) # re-check against the filesystem now
rebuilt = create_processor(profile, source=0) # live camera
rebuilt = create_processor(profile, source="clip.mp4") # recorded frames, same semantics
```
`resolve_bundle(bundle_path, *, calibration_selection=None) -> ResolvedSensorProfile`. The profile
is plain data - no Qt, no application preferences, no device handle - so the same object drives a
camera and a recorded clip. `_set_calibration` on a device affects only its own calibration
selection.
`validate_profile(profile) -> ValidationReport` re-reads the filesystem. It reports
`fingerprint.stale` when a computational asset changed since the profile was resolved.
`create_processor(profile, *, sensor=None, source=None, include=..., contact=...)` returns a
`ProcessingSession` with `start`/`wait`/`latest`/`reset`/`pause`/`resume`/`stop`. It raises
`ProcessorCreationError` naming what is missing rather than returning a session that does nothing.
### Backends
| Identifier | Requires | Produces |
|---|---|---|
| `compound_eye` | `camera`, `stitcher`, `tracker` config | stitched frame, flow, force |
| `single_camera` | `camera` config | frames only - no deformation field, so no force |
| `uv_unroll` | `camera` config, the mapped computational asset | unrolled frame, flow, force |
`known_backends()` and `spec_for(...)` are the registry. `uv_unroll` is not yet materialised from a
profile alone, because it needs geometry, a UV domain, a tactile processor and a capture object
assembled together - that is `orisys.finger.FingerRuntime.open`'s shape. `create_processor` says so
instead of pretending.
## Capabilities
Each capability is decided by the facts it depends on, not by the presence of a file:
| Capability | Unavailable when |
|---|---|
| Video processing | the configuration cannot be loaded |
| Optical flow | no `tracker` block |
| Raw force | the backend produces no deformation field |
| Calibrated force | per axis: the calibration has no curve for it |
| Deformation | the backend computes no deformation field |
| 3D visualization | no scene asset is declared, or the declared one is missing |
Deformation is relative unless a calibration says otherwise; it is reported with its meaning rather
than as a bare "available". `ValidationReport.state` is `ok`, `degraded` (usable, something
optional is missing) or `failed`.
## Calibration
`CalibrationSelection` is `BUNDLE_DEFAULT` (the default), `NONE`, or an explicit path.
- `NONE` is a choice, not a failure: raw force stays available and the calibration fingerprint
records `none`.
- An explicitly selected calibration that is missing or unparseable is an **error**. It is never
replaced by another file.
- Applicability is `declared` when the document carries `meta.sensor_type`, `unknown` when legacy
metadata is absent, or `none`. `unknown` means it applies but its fit is unverified - which is
what the shipped `box` calibration reports.
| Read | Meaning |
|---|---|
| `FrameSnapshot.normal_force_raw` / `shear_x_raw` / `shear_y_raw` | device counts, pre-calibration |
| `FrameSnapshot.normal_force` / `shear_x` / `shear_y` | newtons when a curve applies, counts otherwise |
| `FrameSnapshot.depth`, `centroid`, `contour`, `vnormal` | relative quantities, not newtons |
`Sensor.force_triplets()` and `_force_raw` remain the raw and unfiltered view.
## Fingerprints
`processing_fingerprint` hashes the effective backend, the paired data, and each computational asset's role name with
its content hash. It excludes the bundle location, `display_name`, and every visualization asset: a
relocated or cosmetically renamed bundle keeps its fingerprint, and a missing scene never changes it.
It includes the config minus `name` and `version`.
`calibration_fingerprint` hashes the selected document's content, or the constant `none`.
`profile.inventory()` returns the reproducible snapshot: bundle-relative paths, asset hashes and
both fingerprints. Compare an inventory, not a path, to decide whether two installations would
compute the same numbers.
## Resetting the reference
Both backends process externally supplied frames through the same profile. After a gap in the
input - a pause, a seek, a scene change - call `session.reset()`. It is queued to the stage that
owns the tracker, so the reference frame is rebased on the frame about to be computed rather than
on a buffer the pipeline is still reading.
+62 -47
View File
@@ -1,75 +1,90 @@
# Examples # Examples
Runnable demos live at the `examples/` root. Shared helpers stay in Runnable demos live at the `examples/` root. Shared helpers stay in `pipeline/`, `viz/`, and
`pipeline/`, `viz/`, and `gui/`. Run commands from the repository root. `gui/`. Run commands from the repository root.
## Product Entry Points ## Start here
Both recommended examples create one object, then read a frame at a time. Everything else -
camera reading, stitching, preprocessing, the optical-flow and force stages - happens inside
`StreamPipeline`, and each frame arrives as one `FrameSnapshot`. There is no stage sequence to
remember and no tuple to unpack, and no backend setting to tune.
| Product | Entry | Default config | | Product | Entry | Default config |
|---------|-------|----------------| |---------|-------|----------------|
| ORY-BOX | `basic_pipeline.py` | `box` | | ORY-BOX | `basic_pipeline.py` | `box` |
| ORY-MINI | `mini_pipeline.py` | `mini` | | ORY-MINI | `basic_pipeline.py -c mini` | `mini` |
| ORY-FINGER (2D) | `finger_pipeline.py` | `finger` | | ORY-FINGER | `finger_pipeline.py` | `finger` |
| ORY-FINGER (3D Qt) | `finger_pipeline_3dview.py` | `finger` |
`finger` is the public ORY-FINGER config stem. `dd02` remains a compatibility alias for `dd02-ov`. ```python
# ORY-BOX / ORY-MINI
sensor = orisys.Sensor(0, config_name="box")
with orisys.StreamPipeline(sensor, sensor.get_img, contact=True) as pipe:
while (snap := pipe.wait()) is not None:
print(snap.normal_force, snap.shear_x, snap.shear_y)
## Typical Usage # ORY-FINGER - the runtime owns camera, Blender export, unroll map and valid mask
runtime = FingerRuntime.open(config="finger", video=0)
ORY-BOX: with runtime.pipeline(contact=True) as pipe:
while (snap := pipe.wait()) is not None:
```bash print(snap.normal_force, snap.shear_x, snap.shear_y)
python examples/basic_pipeline.py -v 0 -c box
``` ```
ORY-MINI: `contact=True` runs contact analysis; `include=(...)` chooses which arrays each snapshot carries
(`image`, `flow`, `vnormal`, `vshear`, `depth`, `centroid`, `contour`). Arrays you do not include
```bash are `None`, so you pay only for what you use. `pipe.reset()` resets the tracker safely from your
python examples/mini_pipeline.py -v 0 -c mini loop; `snap.image_size_raw` reports the capture resolution before UV unroll or resize.
```
### Typical usage
ORY-FINGER 2D:
```bash ```bash
python examples/basic_pipeline.py -v 0 -c box # first camera, ORY-BOX
python examples/basic_pipeline.py -v 0 -c mini # ORY-MINI
python examples/basic_pipeline.py -v recording.mp4 -c box
python examples/basic_pipeline.py --no-show # headless: print only
python examples/finger_pipeline.py --camera_id 0 --config finger python examples/finger_pipeline.py --camera_id 0 --config finger
python examples/finger_pipeline.py --video data/dd02-0703/1.mp4
``` ```
ORY-FINGER video file: Keys in both: `q` quits, `r` resets the tracker.
```bash ## The manual form, for when you need the stages
python examples/finger_pipeline.py --video ./data/sample.mp4 --config finger
`examples/archive/` holds the previous versions of these two examples. They call the stages by
hand:
```python
sensor.get_img()
sensor.compute_deformation()
sensor.compute_contact()
fps, flow, vnormal, img, ... = sensor.read_info( ... ) # positional, 13 elements
``` ```
ORY-FINGER 3D Qt dashboard: That path is still supported and still correct - it is the right choice if you are implementing
your own scheduling. It is no longer the recommended starting point, and the archived files carry
a header explaining the difference. They are kept so that code copied from them keeps working.
```bash ## Other root examples
pip install -e ".[gui]"
python examples/finger_pipeline_3dview.py --video 0 --config finger
```
## Other Root Examples - `multi_stream.py` - several sensors at once, using `StreamPipeline` with `StreamRecorder`
- `two_sensor_pipeline.py` - two-sensor multiprocessing demo, manual staging
- `plot_data.py` — single-sensor OpenCV views plus PyQtGraph force charts - `plot_data.py` - single-sensor OpenCV views plus PyQtGraph force charts
- `two_sensor_pipeline.py` — two-sensor multiprocessing demo - `finger_pipeline_3dview.py` - the Qt/open3d dashboard; see the note below
## Requirements ## Requirements
- SDK installed from the repository root, for example `pip install -e .` - SDK installed from the repository root: `pip install -e .`
- camera index or test video path - a camera index or a test video path
- matching config names under `./config/` (or packaged `orisys/configs/`) - matching config names under `./config/` or the packaged `orisys/configs/`
Qt / PyQtGraph examples additionally require: Qt / PyQtGraph examples additionally need `pip install -e ".[gui]"`.
```bash ## The GUI lives here for now
pip install -e ".[gui]"
```
## Notes `examples/gui/` - the 2D viewer, the 3D dashboard and their widgets - is a full Qt + open3d
application rather than an example, and its proper home is the **orisys-studio** application.
That move is **planned but not started**; `finger_pipeline_3dview.py` and `examples/gui/` work
exactly as before. The plan, including the coupling that makes it non-trivial, is in
`docs/plan-gui-extraction.md`.
- `finger_pipeline_3dview.py` is the main interactive finger demo. The practical guidance is only that new SDK work should not build on `examples/gui/`.
It uses `gui/app_paths.py` so the same entry works in editable and packaged runs.
- Dashboard controls: click **启动** to start acquisition; use **Record CSV / Stop CSV**
and the CSV save-directory browser to log contact/force data; `Q` quits, `R` resets,
`S` starts video recording, `E` stops video recording, and `Esc` toggles fullscreen.
- `finger_pipeline.py` is the leanest finger example if you want to
validate UV unroll + force/contact processing without the full dashboard.
+52
View File
@@ -0,0 +1,52 @@
"""Display helpers shared by the pipeline examples.
These exist so an example can show a result without hand-rolling OpenCV arithmetic. Everything
here is presentation only: the SDK never draws, and nothing in ``orisys`` imports this module.
The examples import it through a sibling import, which works when the examples are run from the
examples directory or with ``examples/`` on ``sys.path``.
"""
from __future__ import annotations
import cv2
import numpy as np
def draw_flow_arrows(image, flow, *, threshold=0.4, grid_spacing=10, arrow_scale=1.0,
valid_mask=None):
"""Return the deformation field drawn over ``image``."""
import orisys
return orisys.util.draw_arrows(
image,
flow,
threshold=threshold,
grid_spacing=grid_spacing,
arrow_scale=arrow_scale,
valid_mask=valid_mask,
)
def depth_to_color(depth_map):
"""Normalise a depth/divergence map to 8-bit and apply a colormap for the ``depth`` window.
Returns ``None`` for an empty map, which is what ``read_info`` gives when the field was never
generated, so a caller can skip the window instead of showing saturated noise.
"""
if depth_map is None or not isinstance(depth_map, np.ndarray) or depth_map.size == 0:
return None
lo = float(depth_map.min())
hi = float(depth_map.max())
if hi <= lo:
scaled = np.zeros_like(depth_map, dtype=np.uint8)
else:
scaled = ((depth_map - lo) / (hi - lo) * 255.0).astype(np.uint8)
return cv2.applyColorMap(scaled, cv2.COLORMAP_JET)
def draw_contact(image, contour, centroid):
"""Return ``image`` with the contact region and centroid drawn on it."""
import orisys
return orisys.util.draw_contact(image, contour, centroid)
+159
View File
@@ -0,0 +1,159 @@
"""
ORY-BOX:单传感器基础流程(**已归档的手写版本**)
本示例演示 Orisys SDK 的标准处理流程:
1. 初始化传感器(摄像头或视频文件)
2. 获取图像并计算形变
3. 按需计算接触区域
4. 读取并可视化结果
---
**已归档。请改用 `examples/basic_pipeline.py`。**
保留本文件是为了兼容已经复制过这段代码的用户。它逐级调用
`get_img()` → `compute_deformation()` → `compute_contact()` → `read_info()`,
需要自己记住调用顺序,并按下标解包一个 13 元组。
推荐写法是交给 `orisys.StreamPipeline`:读取、预处理、力/接触解算在内部并发完成,每一帧以
单个 `FrameSnapshot` 交回:
with orisys.StreamPipeline(sensor, sensor.get_img, contact=True) as pipe:
snap = pipe.wait() # 一个对象,一次调用
print(snap.normal_force, snap.shear_x, snap.shear_y)
本文件里的 `sensor.reset()` 在 pipeline 中对应 `pipe.reset()`:它会排队到流程线程执行,
不会与光流阶段争用同一帧缓冲。
**注意**:本文件用中文 `print`,在 cp1252 控制台(未设 UTF-8 的 Windows 终端)会抛
`UnicodeEncodeError`。这是归档前就存在的行为,没有在这里修正——新示例的输出去掉了非 ASCII
字符,因此在任何终端都能跑。
"""
import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import argparse
import cv2
import numpy as np
import orisys
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--video","-v", type=str, default="0", help="视频源:摄像头编号或视频文件路径")
parser.add_argument("--config","-c", type=str, default="box", help="配置文件名称或路径")
parser.add_argument("--verbose","-verbose", type=bool, default=True, help="是否输出详细日志")
parser.add_argument("--cal_path","-cal", type=str, default=None, help="标定文件路径")
args = parser.parse_args()
# 步骤 1:创建传感器对象(必需)
# config_name 可以是内置配置名称,也可以是自定义配置文件路径
# cal_path 为标定文件路径;每台设备建议使用独立的标定文件
# 输入既可以是摄像头编号,也可以是视频文件路径
if args.video.isdigit():
sensor = orisys.Sensor(int(args.video), config_name=args.config, verbose=args.verbose, cal_path=args.cal_path)
else:
sensor = orisys.Sensor(args.video, config_name=args.config, verbose=args.verbose, cal_path=args.cal_path)
print("\n按键说明:'q' 退出程序,'r' 重置追踪器。")
while True:
# 步骤 2:数据处理
# 步骤 2.1:获取并拼接图像(必需)
sensor.get_img()
# 步骤 2.2:计算形变场(必需);返回值表示当前帧是否检测到接触/运动
is_contact_now = sensor.compute_deformation()
# 步骤 2.3:仅在当前帧检测到接触/运动时计算接触区域
# if is_contact_now:
sensor.compute_contact()
# 步骤 2.4:读取结果(FNORMAL/FSHEAR* 在有 sensors/box/force_calibration.json 时为标定后牛顿力;
# *_RAW 始终为原始积分量)
(
fps,
flow,
vnormal,
img,
contour,
centroid,
depth_map,
fnormal,
fshearx,
fsheary,
fnormal_raw,
fshearx_raw,
fsheary_raw,
) = sensor.read_info(
sensor.info.FPS,
sensor.info.VRAW,
sensor.info.VNORMAL,
sensor.info.IMG,
sensor.info.CONTOUR,
sensor.info.CENTROID,
sensor.info.DEPTH,
sensor.info.FNORMAL,
sensor.info.FSHEARX,
sensor.info.FSHEARY,
sensor.info.FNORMAL_RAW,
sensor.info.FSHEARX_RAW,
sensor.info.FSHEARY_RAW,
)
# 步骤 3:可视化显示
# 步骤 3.1:显示形变矢量场
arrows = orisys.util.draw_arrows(
img,
flow,
threshold=2,
grid_spacing=20,
arrow_scale=1.0
)
cv2.imshow("arrow", arrows)
# 步骤 3.2:显示接触区域和质心
image_with_foe = orisys.util.draw_contact(img, contour, centroid)
cv2.imshow("contact", image_with_foe)
# 步骤 3.3:显示深度图
# 将深度值归一化到 0-255 范围
div_abs = depth_map
if div_abs.max() > div_abs.min():
div_normalized = ((div_abs - div_abs.min()) / (div_abs.max() - div_abs.min()) * 255).astype(np.uint8)
else:
div_normalized = np.zeros_like(div_abs, dtype=np.uint8)
# 应用伪彩色映射
cv2.imshow("depth", cv2.applyColorMap(div_normalized, cv2.COLORMAP_JET))
# 步骤 3.4:打印关键结果(标定力 + 原始力)
print(
f"FPS={fps:.2f}, "
f"法向力={fnormal:.4f} N (raw={fnormal_raw:.4f}), "
f"切向力X={fshearx:.4f} N (raw={fshearx_raw:.4f}), "
f"切向力Y={fsheary:.4f} N (raw={fsheary_raw:.4f}), "
f"深度图尺寸={depth_map.shape}, 原始分辨率={sensor.img_size_raw}"
)
# 步骤 3.5:键盘控制
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
print("\n正在退出示例程序...")
break
if key == ord("r"):
# 当光流追踪出现漂移时,可手动重置追踪器
sensor.reset()
print("追踪器已重置。")
# =========================================================================
# 4. 释放资源
# =========================================================================
sensor.disconnect() # 释放视频源及相关资源
if __name__ == '__main__':
main()
+134
View File
@@ -0,0 +1,134 @@
#!/usr/bin/env python3
"""
ORY-FINGER: Minimum UV-unroll finger tactile pipeline. **ARCHIVED — manual version.**
Offline: load Blender export -> build_or_load_domain -> unroll_map_uv.npz
Runtime: camera frame -> FrameUnwarper -> TactileProcessor -> forces/contact
Recommended entry:
python examples/finger_pipeline.py --camera_id 0 --config finger
Current product config: ``finger`` (``dd02`` still resolves to ``dd02-ov``).
---
**ARCHIVED. Use ``examples/finger_pipeline.py`` instead.**
Kept so code copied from this file keeps working. It hand-sequences ``grab_and_process()`` →
``compute_deformation()`` → ``compute_contact()`` → ``read_info()`` and unpacks a ten-element
tuple by position.
The recommended form lets :class:`orisys.StreamPipeline` run those stages concurrently and hand
back one immutable snapshot per frame:
with runtime.pipeline(contact=True, include=("image", "depth")) as pipe:
snap = pipe.wait()
print(snap.normal_force, snap.shear_x, snap.shear_y)
``tactile.reset()`` becomes ``pipe.reset()``, which is queued onto the flow-stage thread instead of
racing it.
This archived version also has a defect: ``--show`` is declared ``store_true`` with
``default=True``, so it can never be turned off. The replacement uses ``--no-show``.
"""
import argparse
import os
import sys
os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0")
import cv2
import orisys
from orisys.finger import FingerRuntime
def main():
parser = argparse.ArgumentParser(description="Minimum UV-unroll finger tactile pipeline")
parser.add_argument("--camera_id", type=int, default=0)
parser.add_argument("--video","-v", help="Video file path (overrides camera)")
parser.add_argument("--config", "-c", default="finger", help="Orisys SDK config name or path")
parser.add_argument("--show", action="store_true", default=True, help="Show unrolled image window")
parser.add_argument("--verbose", action="store_true", help="SDK verbose logs")
args = parser.parse_args()
try:
video = args.camera_id
if args.video:
video = args.video
runtime = FingerRuntime.open(
config=args.config,
video=video,
verbose=args.verbose,
)
except RuntimeError as exc:
print(str(exc), file=sys.stderr)
sys.exit(1)
if args.verbose:
geometry = runtime.geometry
print(
f"UV geometry: native={geometry.native_width}x{geometry.native_height}, "
f"source={geometry.source_width}x{geometry.source_height} "
f"output={geometry.out_width}x{geometry.out_height} "
)
runtime.warmup(color=False)
tactile = runtime.tactile
print("Press 'q' to quit, 'r' to reset tracker.")
try:
while True:
ok, unrolled = runtime.grab_and_process(color=False)
if not ok:
break
tactile.compute_deformation()
tactile.compute_contact()
fps, flow, vnormal, img, contour, centroid, depth_map, fnormal, fshearx, fsheary = tactile.read_info(
tactile.info.FPS,
tactile.info.VRAW,
tactile.info.VNORMAL,
tactile.info.IMG,
tactile.info.CONTOUR,
tactile.info.CENTROID,
tactile.info.DEPTH,
tactile.info.FNORMAL,
tactile.info.FSHEARX,
tactile.info.FSHEARY,
)
status = (
f"FPS={fps:.2f} normal={fnormal:.2f} shear=({fshearx:.2f}, {fsheary:.2f}) "
f"mask={tactile.mask_stats['coverage']:.2f}"
)
if args.show:
arrows = orisys.util.draw_arrows(
unrolled,
flow,
threshold=0.4,
grid_spacing=10,
arrow_scale=1.0,
valid_mask=tactile.valid_mask,
)
cv2.imshow("UV Unrolled", arrows)
print(status, end="\r")
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
print("\nExiting.")
break
if key == ord("r"):
tactile.reset()
print("\nTracker reset.")
finally:
runtime.close()
cv2.destroyAllWindows()
if __name__ == "__main__":
main()
+131 -122
View File
@@ -1,133 +1,142 @@
""" """ORY-BOX / ORY-MINI: read a sensor and get tactile information, in one loop.
ORY-BOX:单传感器基础流程
本示例演示 Orisys SDK 的标准处理流程: The whole recommended flow:
1. 初始化传感器(摄像头或视频文件)
2. 获取图像并计算形变
3. 按需计算接触区域
4. 读取并可视化结果
SDK version: 0.5.0 sensor -> StreamPipeline -> FrameSnapshot per frame
One object reads the camera, pre-processes (stitch, grey, resize), computes the deformation field
and the forces, and hands back one immutable snapshot per frame. There is no stage sequence to
remember and no tuple to unpack by position.
What you do NOT need to touch: the compute backend, the capture queue, the stitch mode, the
low-pass filter, or the order in which the stages run. Those are the SDK's concern and
``StreamPipeline`` sequences them on three concurrent threads. If you later need that control, the
manual form is preserved in ``examples/archive/basic_pipeline_manual.py``.
Run:
python examples/basic_pipeline.py # first camera, box config
python examples/basic_pipeline.py -v 0 -c mini
python examples/basic_pipeline.py -v recording.mp4 -c box
python examples/basic_pipeline.py --no-show # headless: print only
python examples/basic_pipeline.py -v 0 -c box --calibration-dir data/calibrations --device-id BOX-001
Keys: 'q' quit, 'r' reset the tracker.
""" """
import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2
import orisys
import numpy as np
import argparse import argparse
import os
import sys
def main(): # Set before OpenCV is imported: stops the capture backend applying its own transform on top of
parser = argparse.ArgumentParser() # the SDK's preprocessing. This is a capture detail, not a tuning knob you need to own - it is
parser.add_argument("--video","-v", type=str, default="0", help="视频源:摄像头编号或视频文件路径") # here so the file can be copied as a whole.
parser.add_argument("--config","-c", type=str, default="box", help="配置文件名称或路径") os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0")
parser.add_argument("--verbose","-verbose", type=bool, default=True, help="是否输出详细日志")
import cv2
import orisys
from _display import depth_to_color, draw_contact, draw_flow_arrows
def main() -> int:
parser = argparse.ArgumentParser(description="Read a sensor and print its tactile information.")
parser.add_argument("-v", "--video", default="0",
help="camera index or video file path (default: 0)")
parser.add_argument("-c", "--config", default="box",
help="product config: box, mini, finger, or a config path")
calibration = parser.add_mutually_exclusive_group()
calibration.add_argument("--cal", default=None,
help="force calibration file; omit to use the sensor bundle's own")
calibration.add_argument("--calibration-dir", default=None,
help="root containing <product>/<device-id>.json")
parser.add_argument("--device-id", default=None,
help="physical sensor ID; required with --calibration-dir")
parser.add_argument("--no-show", dest="show", action="store_false",
help="do not open the display windows")
parser.add_argument("-verbose", action="store_true", help="SDK verbose logs")
args = parser.parse_args() args = parser.parse_args()
if args.calibration_dir is not None and args.device_id is None:
parser.error("--calibration-dir requires --device-id")
if args.device_id is not None and args.calibration_dir is None and args.cal is None:
parser.error("--device-id requires --calibration-dir or --cal")
# 步骤 1:创建传感器对象(必需) # ------------------------------------------------------------------ create the object
# config_name 可以是内置配置名称,也可以是自定义配置文件路径 # That is the whole setup. The source may be a camera index or a video path.
# cal_path 为标定文件路径;每台设备建议使用独立的标定文件 source = int(args.video) if args.video.isdigit() else args.video
sensor = orisys.Sensor(
# 输入既可以是摄像头编号,也可以是视频文件路径 source, config_name=args.config, cal_path=args.cal,
if args.video.isdigit(): calibration_dir=args.calibration_dir, device_id=args.device_id,
sensor = orisys.Sensor(int(args.video), config_name=args.config, verbose=args.verbose) verbose=args.verbose,
else: )
sensor = orisys.Sensor(args.video, config_name=args.config, verbose=args.verbose) fn_unit, fx_unit, fy_unit = sensor.force_units
selected = sensor.calibration.source_path if sensor.calibration else None
print("\n按键说明:'q' 退出程序,'r' 重置追踪器。") print(f"Force calibration: {selected or 'none'}")
print(f"Force units: FN={fn_unit}, FX={fx_unit}, FY={fy_unit}")
while True: if sensor.device_calibration_missing:
# 步骤 2:数据处理
# 步骤 2.1:获取并拼接图像(必需)
sensor.get_img()
# 步骤 2.2:计算形变场(必需);返回值表示当前帧是否检测到接触/运动
is_contact_now = sensor.compute_deformation()
# 步骤 2.3:仅在当前帧检测到接触/运动时计算接触区域
# if is_contact_now:
sensor.compute_contact()
# 步骤 2.4:读取结果(FNORMAL/FSHEAR* 在有 sensors/box/force_calibration.json 时为标定后牛顿力;
# *_RAW 始终为原始积分量)
(
fps,
flow,
vnormal,
img,
contour,
centroid,
depth_map,
fnormal,
fshearx,
fsheary,
fnormal_raw,
fshearx_raw,
fsheary_raw,
) = sensor.read_info(
sensor.info.FPS,
sensor.info.VRAW,
sensor.info.VNORMAL,
sensor.info.IMG,
sensor.info.CONTOUR,
sensor.info.CENTROID,
sensor.info.DEPTH,
sensor.info.FNORMAL,
sensor.info.FSHEARX,
sensor.info.FSHEARY,
sensor.info.FNORMAL_RAW,
sensor.info.FSHEARX_RAW,
sensor.info.FSHEARY_RAW,
)
# 步骤 3:可视化显示
# 步骤 3.1:显示形变矢量场
arrows = orisys.util.draw_arrows(
img,
flow,
threshold=2,
grid_spacing=20,
arrow_scale=1.0
)
cv2.imshow("arrow", arrows)
# 步骤 3.2:显示接触区域和质心
image_with_foe = orisys.util.draw_contact(img, contour, centroid)
cv2.imshow("contact", image_with_foe)
# 步骤 3.3:显示深度图
# 将深度值归一化到 0-255 范围
div_abs = depth_map
if div_abs.max() > div_abs.min():
div_normalized = ((div_abs - div_abs.min()) / (div_abs.max() - div_abs.min()) * 255).astype(np.uint8)
else:
div_normalized = np.zeros_like(div_abs, dtype=np.uint8)
# 应用伪彩色映射
cv2.imshow("depth", cv2.applyColorMap(div_normalized, cv2.COLORMAP_JET))
# 步骤 3.4:打印关键结果(标定力 + 原始力)
print( print(
f"FPS={fps:.2f}, " f"Warning: no calibration for {args.device_id!r} under "
f"法向力={fnormal:.4f} N (raw={fnormal_raw:.4f}), " f"{args.calibration_dir}; force is raw."
f"切向力X={fshearx:.4f} N (raw={fshearx_raw:.4f}), "
f"切向力Y={fsheary:.4f} N (raw={fsheary_raw:.4f}), "
f"深度图尺寸={depth_map.shape}, 原始分辨率={sensor.img_size_raw}"
) )
# 步骤 3.5:键盘控制 # contact=True runs contact analysis. include=... selects which arrays each snapshot carries;
key = cv2.waitKey(1) & 0xFF # arrays that are not included are None, so you pay only for what you use.
if key == ord("q"): include = ("image", "flow", "depth", "centroid", "contour") if args.show else ("image",)
print("\n正在退出示例程序...")
break if args.show:
if key == ord("r"): print("Keys: 'q' quit, 'r' reset tracker.")
# 当光流追踪出现漂移时,可手动重置追踪器
sensor.reset() try:
print("追踪器已重置。") with orisys.StreamPipeline(sensor, sensor.get_img, contact=True,
include=include) as pipe:
# ========================================================================= while True:
# 4. 释放资源 # ------------------------------------------------------ read one frame
# ========================================================================= # Blocks until a frame newer than the last one is ready. Everything in the
sensor.disconnect() # 释放视频源及相关资源 # snapshot belongs to the same frame.
snap = pipe.wait()
if snap is None:
print("\nInput ended.")
break
# ------------------------------------------------------ use the results
# The *_raw pair is always the unfiltered integrator output.
touching = "yes" if snap.centroid is not None else "no"
fn_unit, fx_unit, fy_unit = sensor.force_units
print(
f"FPS={snap.fps:5.1f} "
f"normal={snap.normal_force:8.4f} {fn_unit} (raw {snap.normal_force_raw:10.2f}) "
f"shear=({snap.shear_x:7.4f} {fx_unit}, {snap.shear_y:7.4f} {fy_unit}) "
f"contact={touching} age={snap.age_ms:4.1f} ms",
end="\r",
)
if args.show and snap.image is not None:
if snap.flow is not None:
cv2.imshow("deformation", draw_flow_arrows(
snap.image, snap.flow, threshold=2, grid_spacing=20))
if snap.contour is not None and snap.centroid is not None:
cv2.imshow("contact", draw_contact(
snap.image, snap.contour, snap.centroid))
colour = depth_to_color(snap.depth)
if colour is not None:
cv2.imshow("depth", colour)
key = cv2.waitKey(1) & 0xFF if args.show else -1
if key == ord("q"):
print("\nExiting.")
break
if key == ord("r"):
# Queued onto the flow stage, so this is safe from the consumer loop.
pipe.reset()
print("\nTracker reset.")
except KeyboardInterrupt:
print("\nInterrupted.")
finally:
sensor.disconnect()
if args.show:
cv2.destroyAllWindows()
return 0
if __name__ == '__main__': if __name__ == "__main__":
main() sys.exit(main())
+105
View File
@@ -0,0 +1,105 @@
# 滤波预设:可直接复制的 `filters` 段
`json.load` 不支持注释,所以这个目录就是"在配置里写说明"的替代品:**把下面任意一段整块替换**
到 `sensors/<name>/sensor.json` 的 `"filters"` 段即可。原有的 `filters` 段请整段替换,
不要只改其中一行(避免出现 `preset` 与数值混写的歧义——显式键优先于预设值,混写容易看错)。
配置里**没写**的滤波键等于"关":整段删掉就是什么都不滤,和 `off` 那段等价。
背景、代价与可追溯性约定见 `docs/filters.md`;延迟/噪声实测见
`D:\orisys_ws\docs\performance\orisys-sdk-perf-plan.md`(原 `_perf/` 已从本仓库移除:幸存脚本与
片源在 `D:\orisys_ws\docs\performance\perf-measurements\`,该文档保留全部实测数字)。
下面每一段都有一个同目录的机器可读副本 `filters_<预设名>.json`(四个文件),
`tests/test_filter_presets.py` 会核对它们与 `FILTER_PRESETS` 完全一致 —— 所以改预设时
这些文件会一起红,不会悄悄过时。
---
## `off` —— 原始读数(默认)
```json
"filters": {
"flow_ema_alpha": 1.0,
"force_median_window": 1,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
```
## `low_latency` —— 只去尖峰(≈1 帧 @30fps)
```json
"filters": {
"flow_ema_alpha": 1.0,
"force_median_window": 3,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
```
## `balanced` —— 建议默认(≈2 帧 @30fps)
```json
"filters": {
"flow_ema_alpha": 0.5,
"force_median_window": 3,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
```
## `low_noise` —— 底噪优先(≈3 帧 @30fps)
```json
"filters": {
"flow_ema_alpha": 0.3,
"force_median_window": 3,
"force_lowpass_cutoff_hz": 10.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
```
## 用预设名(等价写法,数值由代码展开)
```json
"filters": { "preset": "balanced" }
```
`off / low_latency / balanced / low_noise` 四个名字来自
`src/orisys/sensor.py::FILTER_PRESETS`(单一真值);想从预设出发只改一个旋钮,就在同一段里写
那一个键,例如:
```json
"filters": { "preset": "balanced", "force_lowpass_cutoff_hz": 10.0 }
```
---
## 复制后请核对"实际生效值"
配置只是**意图**;`Sensor.configure()` 可以在运行时再改,所以审计与记录一律看生效值:
```python
from orisys.sensor import Sensor
s = Sensor("external", config_name="mini", verbose=False) # 无需硬件即可核对
print(s.filters_state()["preset"], s.filters_state()["measurement"], s.filters_state()["readout"])
```
`flow_ema_applied` / `force_median_applied` / `force_lowpass_applied` 三个布尔就是
"这一级到底有没有在滤"的答案;`flow_ema_alpha` 属于 **measurement**(改变测量,
不同值的两次录制不可比),中值与低通属于 **readout**(只影响读数呈现)。
导出标定/写记录时把同一个状态交给指纹函数,文件里就会留下可核对的设置:
```python
from orisys.calibration import pipeline_from_sensor_json
pipeline = pipeline_from_sensor_json("sensors/mini/sensor.json", filters_state=s.filters_state())
print(pipeline["filters"]) # measurement / readout / preset / source
```
详见 `docs/filters.md` 第 7 节。
+7
View File
@@ -0,0 +1,7 @@
{
"flow_ema_alpha": 0.5,
"force_median_window": 3,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
@@ -0,0 +1,7 @@
{
"flow_ema_alpha": 1.0,
"force_median_window": 3,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
+7
View File
@@ -0,0 +1,7 @@
{
"flow_ema_alpha": 0.3,
"force_median_window": 3,
"force_lowpass_cutoff_hz": 10.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
+7
View File
@@ -0,0 +1,7 @@
{
"flow_ema_alpha": 1.0,
"force_median_window": 1,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}
+74 -77
View File
@@ -1,14 +1,21 @@
#!/usr/bin/env python3 """ORY-FINGER: UV-unrolled finger tactile pipeline.
"""
ORY-FINGER:Minimum UV-unroll finger tactile pipeline.
Offline: load Blender export -> build_or_load_domain -> unroll_map_uv.npz Same shape as ``basic_pipeline.py`` - one object, one call per frame - with the finger-specific
Runtime: camera frame -> FrameUnwarper -> TactileProcessor -> forces/contact capture path underneath:
Recommended entry: FingerRuntime -> runtime.pipeline() -> FrameSnapshot per frame
python examples/finger_pipeline.py --camera_id 0 --config finger
Current product config: ``finger`` (``dd02`` still resolves to ``dd02-ov``). ``FingerRuntime`` owns the camera, the Blender export (mesh, intrinsics, world-to-camera), the UV
unroll map and the valid-region mask. ``runtime.pipeline()`` runs those stages concurrently and
returns one immutable snapshot per frame, so there is nothing to sequence and no tuple to unpack.
Run:
python examples/finger_pipeline.py --camera_id 0 --config finger
python examples/finger_pipeline.py --video path/to/clip.mp4
python examples/finger_pipeline.py --no-show # headless: print only
Keys: 'q' quit, 'r' reset the tracker.
""" """
import argparse import argparse
@@ -18,95 +25,85 @@ import sys
os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0") os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0")
import cv2 import cv2
import orisys
from _display import draw_flow_arrows
from orisys.finger import FingerRuntime from orisys.finger import FingerRuntime
def main(): def main() -> int:
parser = argparse.ArgumentParser(description="Minimum UV-unroll finger tactile pipeline") parser = argparse.ArgumentParser(description="UV-unrolled finger tactile pipeline")
parser.add_argument("--camera_id", type=int, default=0) parser.add_argument("--camera_id", type=int, default=0, help="camera index (default 0)")
parser.add_argument("--video","-v", help="Video file path (overrides camera)") parser.add_argument("-v", "--video", help="video file path; overrides --camera_id")
parser.add_argument("--config", "-c", default="finger", help="Orisys SDK config name or path") parser.add_argument("-c", "--config", default="finger",
parser.add_argument("--show", action="store_true", default=True, help="Show unrolled image window") help="product config name or path (default: finger)")
parser.add_argument("--verbose", action="store_true", help="SDK verbose logs") parser.add_argument("--no-show", dest="show", action="store_false",
help="do not open the display window")
parser.add_argument("-verbose", action="store_true", help="SDK verbose logs")
args = parser.parse_args() args = parser.parse_args()
source = args.video if args.video else args.camera_id
# ------------------------------------------------------------------ create the object
try: try:
video = args.camera_id runtime = FingerRuntime.open(config=args.config, video=source, verbose=args.verbose)
if args.video:
video = args.video
runtime = FingerRuntime.open(
config=args.config,
video=video,
verbose=args.verbose,
)
except RuntimeError as exc: except RuntimeError as exc:
print(str(exc), file=sys.stderr) print(str(exc), file=sys.stderr)
sys.exit(1) return 1
if args.verbose: if args.verbose:
geometry = runtime.geometry g = runtime.geometry
print( print(f"UV geometry: native={g.native_width}x{g.native_height} "
f"UV geometry: native={geometry.native_width}x{geometry.native_height}, " f"source={g.source_width}x{g.source_height} "
f"source={geometry.source_width}x{geometry.source_height} " f"output={g.out_width}x{g.out_height}")
f"output={geometry.out_width}x{geometry.out_height} "
)
# Fill the flow/HHD caches once so the first published frame is not a warm-up outlier.
runtime.warmup(color=False) runtime.warmup(color=False)
tactile = runtime.tactile
print("Press 'q' to quit, 'r' to reset tracker.") if args.show:
print("Keys: 'q' quit, 'r' reset tracker.")
include = ("image", "flow") if args.show else ("image",)
try: try:
while True: with runtime.pipeline(contact=True, include=include) as pipe:
ok, unrolled = runtime.grab_and_process(color=False) while True:
if not ok: # ------------------------------------------------------ read one frame
break snap = pipe.wait()
if snap is None:
print("\nInput ended.")
break
tactile.compute_deformation() # ------------------------------------------------------ use the results
tactile.compute_contact() coverage = runtime.tactile.mask_stats.get("coverage")
coverage_text = f"{coverage:.2f}" if isinstance(coverage, float) else "n/a"
fps, flow, vnormal, img, contour, centroid, depth_map, fnormal, fshearx, fsheary = tactile.read_info( fn_unit, fx_unit, fy_unit = runtime.tactile.force_units
tactile.info.FPS, print(
tactile.info.VRAW, f"FPS={snap.fps:5.1f} "
tactile.info.VNORMAL, f"normal={snap.normal_force:8.4f} {fn_unit} "
tactile.info.IMG, f"shear=({snap.shear_x:7.4f} {fx_unit}, {snap.shear_y:7.4f} {fy_unit}) "
tactile.info.CONTOUR, f"mask={coverage_text} age={snap.age_ms:4.1f} ms",
tactile.info.CENTROID, end="\r",
tactile.info.DEPTH,
tactile.info.FNORMAL,
tactile.info.FSHEARX,
tactile.info.FSHEARY,
)
status = (
f"FPS={fps:.2f} normal={fnormal:.2f} shear=({fshearx:.2f}, {fsheary:.2f}) "
f"mask={tactile.mask_stats['coverage']:.2f}"
)
if args.show:
arrows = orisys.util.draw_arrows(
unrolled,
flow,
threshold=0.4,
grid_spacing=10,
arrow_scale=1.0,
valid_mask=tactile.valid_mask,
) )
cv2.imshow("UV Unrolled", arrows)
print(status, end="\r") if args.show and snap.image is not None and snap.flow is not None:
cv2.imshow("tactile (UV unrolled)", draw_flow_arrows(
snap.image, snap.flow, threshold=0.4, grid_spacing=10,
valid_mask=runtime.tactile.valid_mask))
key = cv2.waitKey(1) & 0xFF key = cv2.waitKey(1) & 0xFF
if key == ord("q"): if key == ord("q"):
print("\nExiting.") print("\nExiting.")
break break
if key == ord("r"): if key == ord("r"):
tactile.reset() pipe.reset()
print("\nTracker reset.") print("\nTracker reset.")
except KeyboardInterrupt:
print("\nInterrupted.")
finally: finally:
runtime.close() runtime.close()
cv2.destroyAllWindows() if args.show:
cv2.destroyAllWindows()
return 0
if __name__ == "__main__": if __name__ == "__main__":
main() sys.exit(main())
+10 -9
View File
@@ -9,9 +9,10 @@ Acquisition adapters (no widgets) live in ``gui.sensors``.
Shared bootstrap: ``gui.app``, ``gui.app_paths``, ``gui.log_config``, ``gui.widgets``. Shared bootstrap: ``gui.app``, ``gui.app_paths``, ``gui.log_config``, ``gui.widgets``.
""" """
from viz.contact_arrow import ContactArrowStyle
from .log_config import LOG_CONFIG, LogConfig from .log_config import LOG_CONFIG, LogConfig
from .viewer3d import ( from .viewer3d import (
FingerDashboardWindow,
LAYOUT, LAYOUT,
LOG_PANEL, LOG_PANEL,
MESH_CONTACT_ARROW, MESH_CONTACT_ARROW,
@@ -19,33 +20,33 @@ from .viewer3d import (
PANEL_SETUP, PANEL_SETUP,
TOP_ROW, TOP_ROW,
DashboardLayout, DashboardLayout,
FingerDashboardWindow,
MeshLighting, MeshLighting,
PanelRow, PanelRow,
PanelSetup, PanelSetup,
PanelSlot, PanelSlot,
run_dashboard, run_dashboard,
top_row_splitter_sizes,
top_row_slot_index, top_row_slot_index,
top_row_splitter_sizes,
) )
from viz.contact_arrow import ContactArrowStyle
__all__ = [ __all__ = [
"DashboardLayout",
"FingerDashboardWindow",
"LAYOUT", "LAYOUT",
"LOG_CONFIG", "LOG_CONFIG",
"LOG_PANEL", "LOG_PANEL",
"LogConfig",
"MESH_CONTACT_ARROW", "MESH_CONTACT_ARROW",
"MESH_LIGHTING", "MESH_LIGHTING",
"ContactArrowStyle",
"MeshLighting",
"PANEL_SETUP", "PANEL_SETUP",
"TOP_ROW", "TOP_ROW",
"ContactArrowStyle",
"DashboardLayout",
"FingerDashboardWindow",
"LogConfig",
"MeshLighting",
"PanelRow", "PanelRow",
"PanelSetup", "PanelSetup",
"PanelSlot", "PanelSlot",
"run_dashboard", "run_dashboard",
"top_row_splitter_sizes",
"top_row_slot_index", "top_row_slot_index",
"top_row_splitter_sizes",
] ]
+1 -1
View File
@@ -34,8 +34,8 @@ def default_config_path() -> str:
def require_gui_extras() -> None: def require_gui_extras() -> None:
try: try:
import open3d # noqa: F401 import open3d # noqa: F401
from PyQt5 import QtWidgets # noqa: F401
import pyqtgraph # noqa: F401 import pyqtgraph # noqa: F401
from PyQt5 import QtWidgets # noqa: F401
except ImportError as exc: except ImportError as exc:
raise SystemExit('GUI example requires: pip install "orisys[gui]"') from exc raise SystemExit('GUI example requires: pip install "orisys[gui]"') from exc
+3 -3
View File
@@ -1,23 +1,23 @@
"""Sensor acquisition adapters for Qt / multi-sensor viewers (no Qt widgets).""" """Sensor acquisition adapters for Qt / multi-sensor viewers (no Qt widgets)."""
from . import roi_geometry
from .adapter import ( from .adapter import (
Dd06RoiAdapter, Dd06RoiAdapter,
FrameSnapshot, FrameSnapshot,
OrisysSensorAdapter, OrisysSensorAdapter,
SensorAdapter, SensorAdapter,
_snapshot_from_sensor,
calibrate_normal_force, calibrate_normal_force,
create_sensor_adapter, create_sensor_adapter,
_snapshot_from_sensor,
) )
from . import roi_geometry
__all__ = [ __all__ = [
"Dd06RoiAdapter", "Dd06RoiAdapter",
"FrameSnapshot", "FrameSnapshot",
"OrisysSensorAdapter", "OrisysSensorAdapter",
"SensorAdapter", "SensorAdapter",
"_snapshot_from_sensor",
"calibrate_normal_force", "calibrate_normal_force",
"create_sensor_adapter", "create_sensor_adapter",
"roi_geometry", "roi_geometry",
"_snapshot_from_sensor",
] ]
+14 -15
View File
@@ -5,11 +5,11 @@ from __future__ import annotations
import time import time
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from typing import Any
from typing import Any, Optional
import cv2 import cv2
import numpy as np import numpy as np
import orisys import orisys
from .roi_geometry import ( from .roi_geometry import (
@@ -21,7 +21,6 @@ from .roi_geometry import (
warp_roi_to_rectangle, warp_roi_to_rectangle,
) )
NORMAL_FORCE_CAL_SLOPE = 3.92685554668795e-05 NORMAL_FORCE_CAL_SLOPE = 3.92685554668795e-05
NORMAL_FORCE_CAL_OFFSET = 0.0217596967278535 NORMAL_FORCE_CAL_OFFSET = 0.0217596967278535
@@ -30,10 +29,10 @@ NORMAL_FORCE_CAL_OFFSET = 0.0217596967278535
class FrameSnapshot: class FrameSnapshot:
"""Normalized per-frame payload consumed by the Qt viewer.""" """Normalized per-frame payload consumed by the Qt viewer."""
image: Optional[np.ndarray] = None image: np.ndarray | None = None
flow: Optional[np.ndarray] = None flow: np.ndarray | None = None
depth: Optional[np.ndarray] = None depth: np.ndarray | None = None
contour: Optional[np.ndarray] = None contour: np.ndarray | None = None
centroid: Any = None centroid: Any = None
# Raw SDK force channels used for curves (same units for FN/FX/FY). # Raw SDK force channels used for curves (same units for FN/FX/FY).
normal_force: float = 0.0 normal_force: float = 0.0
@@ -43,10 +42,10 @@ class FrameSnapshot:
normal_force_calibrated: float = 0.0 normal_force_calibrated: float = 0.0
fps: float = 0.0 fps: float = 0.0
timestamp: float = 0.0 timestamp: float = 0.0
raw_image: Optional[np.ndarray] = None raw_image: np.ndarray | None = None
processed_image: Optional[np.ndarray] = None processed_image: np.ndarray | None = None
flow_overlay: Optional[np.ndarray] = None flow_overlay: np.ndarray | None = None
record_frame: Optional[np.ndarray] = None record_frame: np.ndarray | None = None
def calibrate_normal_force(raw_value: float) -> float: def calibrate_normal_force(raw_value: float) -> float:
@@ -62,7 +61,7 @@ class SensorAdapter(ABC):
raise NotImplementedError raise NotImplementedError
@abstractmethod @abstractmethod
def read(self) -> Optional[FrameSnapshot]: def read(self) -> FrameSnapshot | None:
raise NotImplementedError raise NotImplementedError
@abstractmethod @abstractmethod
@@ -89,7 +88,7 @@ def _snapshot_from_sensor(
*, *,
raw_image=None, raw_image=None,
processed_image=None, processed_image=None,
) -> Optional[FrameSnapshot]: ) -> FrameSnapshot | None:
"""Build a FrameSnapshot with HHD/forces refreshed every frame.""" """Build a FrameSnapshot with HHD/forces refreshed every frame."""
if sensor is None or sensor.img is None: if sensor is None or sensor.img is None:
return None return None
@@ -186,7 +185,7 @@ class OrisysSensorAdapter(SensorAdapter):
verbose=self.verbose, verbose=self.verbose,
) )
def read(self) -> Optional[FrameSnapshot]: def read(self) -> FrameSnapshot | None:
if self.sensor is None: if self.sensor is None:
return None return None
img = self.sensor.get_img() img = self.sensor.get_img()
@@ -255,7 +254,7 @@ class Dd06RoiAdapter(SensorAdapter):
f"video={self._source}" f"video={self._source}"
) )
def read(self) -> Optional[FrameSnapshot]: def read(self) -> FrameSnapshot | None:
if self.sensor is None or self.cap is None: if self.sensor is None or self.cap is None:
return None return None
+17 -16
View File
@@ -8,13 +8,14 @@ Does not import camera, ROI, or `orisys.Sensor`.
from __future__ import annotations from __future__ import annotations
from dataclasses import dataclass from dataclasses import dataclass
from typing import Literal, Optional, Sequence, Tuple from typing import Literal
import cv2 import cv2
import numpy as np import numpy as np
import orisys
from scipy.signal import butter, filtfilt from scipy.signal import butter, filtfilt
import orisys
FORCE_SAMPLE_RATE_HZ = 30.0 FORCE_SAMPLE_RATE_HZ = 30.0
PLOT_BUFFER_POINTS = 120 PLOT_BUFFER_POINTS = 120
PLOT_WINDOW_SEC = PLOT_BUFFER_POINTS / FORCE_SAMPLE_RATE_HZ PLOT_WINDOW_SEC = PLOT_BUFFER_POINTS / FORCE_SAMPLE_RATE_HZ
@@ -40,14 +41,14 @@ class ViewerFrame:
image_height: int image_height: int
image_width: int image_width: int
aspect_ratio: float # width / height aspect_ratio: float # width / height
depth: Optional[np.ndarray] depth: np.ndarray | None
normal_force: float normal_force: float
shear_x: float shear_x: float
shear_y: float shear_y: float
normal_force_calibrated: float normal_force_calibrated: float
fps: float fps: float
timestamp: float timestamp: float
record_frame: Optional[np.ndarray] record_frame: np.ndarray | None
@dataclass @dataclass
@@ -57,8 +58,8 @@ class ForcePlotState:
times: np.ndarray # shape (n,) times: np.ndarray # shape (n,)
values: np.ndarray # shape (n, 3) FN, FX, FY values: np.ndarray # shape (n, 3) FN, FX, FY
sample_count: int sample_count: int
x_range: Tuple[float, float] x_range: tuple[float, float]
y_ranges: Tuple[Tuple[float, float], Tuple[float, float], Tuple[float, float]] y_ranges: tuple[tuple[float, float], tuple[float, float], tuple[float, float]]
def lowpass_filter( def lowpass_filter(
@@ -87,7 +88,7 @@ def lowpass_filter(
return filtfilt(b, a, y) return filtfilt(b, a, y)
def ensure_bgr(img: Optional[np.ndarray]) -> Optional[np.ndarray]: def ensure_bgr(img: np.ndarray | None) -> np.ndarray | None:
if img is None: if img is None:
return None return None
if img.ndim == 2: if img.ndim == 2:
@@ -97,14 +98,14 @@ def ensure_bgr(img: Optional[np.ndarray]) -> Optional[np.ndarray]:
return img return img
def image_shape_meta(img: np.ndarray) -> Tuple[int, int, float]: def image_shape_meta(img: np.ndarray) -> tuple[int, int, float]:
"""Return (height, width, width/height) from a display image.""" """Return (height, width, width/height) from a display image."""
h, w = int(img.shape[0]), int(img.shape[1]) h, w = int(img.shape[0]), int(img.shape[1])
aspect = float(w) / float(h) if h > 0 else 1.0 aspect = float(w) / float(h) if h > 0 else 1.0
return h, w, aspect return h, w, aspect
def _base_bgr(snapshot) -> Optional[np.ndarray]: def _base_bgr(snapshot) -> np.ndarray | None:
for attr in ("image", "processed_image", "raw_image"): for attr in ("image", "processed_image", "raw_image"):
out = ensure_bgr(getattr(snapshot, attr, None)) out = ensure_bgr(getattr(snapshot, attr, None))
if out is not None: if out is not None:
@@ -152,7 +153,7 @@ def build_display_image(
return select_display_image(snapshot) return select_display_image(snapshot)
def bgr_to_rgb_bytes(img_bgr: np.ndarray) -> Tuple[np.ndarray, int, int]: def bgr_to_rgb_bytes(img_bgr: np.ndarray) -> tuple[np.ndarray, int, int]:
"""Convert BGR uint8 image to contiguous RGB for QImage.""" """Convert BGR uint8 image to contiguous RGB for QImage."""
rgb = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB) rgb = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB)
if not rgb.flags["C_CONTIGUOUS"]: if not rgb.flags["C_CONTIGUOUS"]:
@@ -231,7 +232,7 @@ class ForceHistory:
self.data[-1, 0, :] = t self.data[-1, 0, :] = t
self.data[-1, 1, :] = values self.data[-1, 1, :] = values
def _smooth_y_range(self, channel: int, ymin: float, ymax: float) -> Tuple[float, float]: def _smooth_y_range(self, channel: int, ymin: float, ymax: float) -> tuple[float, float]:
if self._y_range_smooth[channel] is None: if self._y_range_smooth[channel] is None:
smoothed = (ymin, ymax) smoothed = (ymin, ymax)
else: else:
@@ -251,7 +252,7 @@ class ForceHistory:
pad = max((smoothed[1] - smoothed[0]) * 0.12, 0.01) pad = max((smoothed[1] - smoothed[0]) * 0.12, 0.01)
return smoothed[0] - pad, smoothed[1] + pad return smoothed[0] - pad, smoothed[1] + pad
def plot_state(self) -> Optional[ForcePlotState]: def plot_state(self) -> ForcePlotState | None:
n = self.sample_count n = self.sample_count
if n == 0: if n == 0:
return None return None
@@ -305,8 +306,8 @@ class QtViewerBackend:
lowpass_cutoff_hz=lowpass_cutoff_hz, lowpass_cutoff_hz=lowpass_cutoff_hz,
y_axis_min_span=y_axis_min_span, y_axis_min_span=y_axis_min_span,
) )
self._start_time: Optional[float] = None self._start_time: float | None = None
self.last_viewer_frame: Optional[ViewerFrame] = None self.last_viewer_frame: ViewerFrame | None = None
self.arrow_background: ArrowBackground = arrow_background self.arrow_background: ArrowBackground = arrow_background
def set_arrow_background(self, background: ArrowBackground) -> None: def set_arrow_background(self, background: ArrowBackground) -> None:
@@ -357,7 +358,7 @@ class QtViewerBackend:
self._start_time = None self._start_time = None
self.last_viewer_frame = None self.last_viewer_frame = None
def ingest(self, snapshot, *, now: Optional[float] = None) -> ViewerFrame: def ingest(self, snapshot, *, now: float | None = None) -> ViewerFrame:
"""Convert a FrameSnapshot into ViewerFrame and update force history.""" """Convert a FrameSnapshot into ViewerFrame and update force history."""
display = build_display_image(snapshot, self.arrow_background) display = build_display_image(snapshot, self.arrow_background)
h, w, aspect = image_shape_meta(display) h, w, aspect = image_shape_meta(display)
@@ -396,5 +397,5 @@ class QtViewerBackend:
self.last_viewer_frame = frame self.last_viewer_frame = frame
return frame return frame
def plot_state(self) -> Optional[ForcePlotState]: def plot_state(self) -> ForcePlotState | None:
return self.force_history.plot_state() return self.force_history.plot_state()
+6 -6
View File
@@ -2,8 +2,8 @@
from .dashboard import FingerDashboardWindow, run_dashboard from .dashboard import FingerDashboardWindow, run_dashboard
from .layout import ( from .layout import (
LOG_PANEL,
LAYOUT, LAYOUT,
LOG_PANEL,
MESH_CONTACT_ARROW, MESH_CONTACT_ARROW,
MESH_LIGHTING, MESH_LIGHTING,
PANEL_SETUP, PANEL_SETUP,
@@ -13,24 +13,24 @@ from .layout import (
PanelRow, PanelRow,
PanelSetup, PanelSetup,
PanelSlot, PanelSlot,
top_row_splitter_sizes,
top_row_slot_index, top_row_slot_index,
top_row_splitter_sizes,
) )
__all__ = [ __all__ = [
"DashboardLayout",
"FingerDashboardWindow",
"LAYOUT", "LAYOUT",
"LOG_PANEL", "LOG_PANEL",
"MESH_CONTACT_ARROW", "MESH_CONTACT_ARROW",
"MESH_LIGHTING", "MESH_LIGHTING",
"MeshLighting",
"PANEL_SETUP", "PANEL_SETUP",
"TOP_ROW", "TOP_ROW",
"DashboardLayout",
"FingerDashboardWindow",
"MeshLighting",
"PanelRow", "PanelRow",
"PanelSetup", "PanelSetup",
"PanelSlot", "PanelSlot",
"run_dashboard", "run_dashboard",
"top_row_splitter_sizes",
"top_row_slot_index", "top_row_slot_index",
"top_row_splitter_sizes",
] ]
+25 -4
View File
@@ -3,6 +3,7 @@
from __future__ import annotations from __future__ import annotations
import contextlib import contextlib
import logging
import os import os
import sys import sys
from dataclasses import dataclass from dataclasses import dataclass
@@ -11,6 +12,8 @@ import cv2
DEFAULT_MAX_CAMERA_INDEX = 10 DEFAULT_MAX_CAMERA_INDEX = 10
logger = logging.getLogger(__name__)
@dataclass(frozen=True) @dataclass(frozen=True)
class CameraInfo: class CameraInfo:
@@ -62,8 +65,14 @@ def _suppress_opencv_noise():
if set_level is not None and prev_level is not None: if set_level is not None and prev_level is not None:
try: try:
set_level(prev_level) set_level(prev_level)
except Exception: except Exception as exc:
pass # Best-effort: a failure only leaves OpenCV probe noise suppressed,
# but report it so a stuck log level is diagnosable.
logger.warning(
"camera discovery: could not restore OpenCV log level %r: %s",
prev_level,
exc,
)
def _backend_candidates() -> list[tuple[str, int]]: def _backend_candidates() -> list[tuple[str, int]]:
@@ -115,9 +124,21 @@ def list_cameras(max_index: int = DEFAULT_MAX_CAMERA_INDEX) -> list[CameraInfo]:
backends = _backend_candidates() backends = _backend_candidates()
with _suppress_opencv_noise(): with _suppress_opencv_noise():
for index in range(0, max_index + 1): for index in range(max_index + 1):
for backend_name, backend_api in backends: for backend_name, backend_api in backends:
info = _try_open(index, backend_name, backend_api) try:
info = _try_open(index, backend_name, backend_api)
except Exception as exc:
# Same hazard as the SDK enumerator: an escaping error for one
# backend would abort the whole probe and hide every other
# camera. Skip that candidate and keep going.
logger.warning(
"camera discovery: probing index %d via %s failed: %s",
index,
backend_name,
exc,
)
continue
if info is None: if info is None:
continue continue
if index in seen_indices: if index in seen_indices:
+7 -10
View File
@@ -6,6 +6,9 @@ import math
import sys import sys
import time import time
from pipeline.config import PipelineConfig
from pipeline.csv_recorder import CsvRecorder
from pipeline.service import FingerPipeline
from PyQt5.QtCore import Qt, QTimer from PyQt5.QtCore import Qt, QTimer
from PyQt5.QtGui import QIcon from PyQt5.QtGui import QIcon
from PyQt5.QtWidgets import ( from PyQt5.QtWidgets import (
@@ -17,23 +20,17 @@ from PyQt5.QtWidgets import (
QWidget, QWidget,
) )
from pipeline.config import PipelineConfig
from pipeline.csv_recorder import CsvRecorder
from pipeline.service import FingerPipeline
from gui.app_paths import resource_path from gui.app_paths import resource_path
from gui.widgets import TeeStream, WidgetLogHandler, make_log_widget
from .camera_discovery import list_cameras, pick_default_camera from .camera_discovery import list_cameras, pick_default_camera
from .preferences import get_preferred_video_source, load_preferences, save_preferences
from .layout import ( from .layout import (
DashboardLayout,
LAYOUT, LAYOUT,
PANEL_SETUP, DashboardLayout,
PanelSetup,
top_row_splitter_sizes,
) )
from .mesh_view_state import MeshViewState from .mesh_view_state import MeshViewState
from .preferences import get_preferred_video_source, load_preferences, save_preferences
from .styles import STYLE from .styles import STYLE
from gui.widgets import TeeStream, WidgetLogHandler, make_log_widget
from .widgets import ( from .widgets import (
ArrowPanelWidget, ArrowPanelWidget,
ForceChartWidget, ForceChartWidget,
+3 -2
View File
@@ -47,11 +47,12 @@ import math
from dataclasses import dataclass from dataclasses import dataclass
from typing import Literal from typing import Literal
from gui.log_config import LOG_CONFIG, LogConfig from viz.contact_arrow import ContactArrowStyle
from viz.contact_arrow import ContactArrowStyle, DEFAULT_CONTACT_ARROW_STYLE
from viz.flow_arrow import FlowArrowStyle from viz.flow_arrow import FlowArrowStyle
from viz.render_2d import ArrowBackground, ArrowCanvasColors from viz.render_2d import ArrowBackground, ArrowCanvasColors
from gui.log_config import LOG_CONFIG, LogConfig
PanelId = Literal["mesh", "arrows", "charts", "log", "settings"] PanelId = Literal["mesh", "arrows", "charts", "log", "settings"]
+1 -1
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from PyQt5.QtCore import QObject, pyqtSignal from PyQt5.QtCore import QObject, pyqtSignal
from .layout import DashboardLayout, LAYOUT from .layout import LAYOUT, DashboardLayout
class MeshViewState(QObject): class MeshViewState(QObject):
+3 -3
View File
@@ -3,14 +3,14 @@
from __future__ import annotations from __future__ import annotations
import cv2 import cv2
from pipeline.frame import FingerFrameView from pipeline.frame import FingerFrameView
from viz.render_2d import ArrowBackground, ArrowCanvasColors, render_uv_arrows from viz.render_2d import ArrowBackground, ArrowCanvasColors, render_uv_arrows
from ..layout import DashboardLayout, LAYOUT
from .image_panel import ImagePanel
from gui.widgets.qt_pixmap import ndarray_to_pixmap from gui.widgets.qt_pixmap import ndarray_to_pixmap
from ..layout import LAYOUT, DashboardLayout
from .image_panel import ImagePanel
class ArrowPanelWidget(ImagePanel): class ArrowPanelWidget(ImagePanel):
"""UV unrolled flow-arrow panel.""" """UV unrolled flow-arrow panel."""
@@ -3,14 +3,14 @@
from __future__ import annotations from __future__ import annotations
import numpy as np import numpy as np
from pipeline.frame import FingerFrameView
from PyQt5.QtGui import QFont from PyQt5.QtGui import QFont
from PyQt5.QtWidgets import QSizePolicy from PyQt5.QtWidgets import QSizePolicy
from ..layout import DashboardLayout, LAYOUT
from ..styles import LOG_FONT_PRIMARY, UI_FONT_PRIMARY
from pipeline.frame import FingerFrameView
from viz.signal_filters import lowpass_filter from viz.signal_filters import lowpass_filter
from ..layout import LAYOUT, DashboardLayout
from ..styles import LOG_FONT_PRIMARY, UI_FONT_PRIMARY
def _require_pyqtgraph(): def _require_pyqtgraph():
try: try:
@@ -2,16 +2,16 @@
from __future__ import annotations from __future__ import annotations
from pipeline.frame import FingerFrameView
from PyQt5.QtCore import QEvent, Qt from PyQt5.QtCore import QEvent, Qt
from PyQt5.QtGui import QMouseEvent from PyQt5.QtGui import QMouseEvent
from pipeline.frame import FingerFrameView
from viz.offscreen_mesh import OffscreenFingerMeshRenderer from viz.offscreen_mesh import OffscreenFingerMeshRenderer
from ..layout import DashboardLayout, LAYOUT
from ..mesh_view_state import MeshViewState
from ..mesh_render_worker import MeshRenderWorker
from gui.widgets.qt_pixmap import ndarray_to_pixmap from gui.widgets.qt_pixmap import ndarray_to_pixmap
from ..layout import LAYOUT, DashboardLayout
from ..mesh_render_worker import MeshRenderWorker
from ..mesh_view_state import MeshViewState
from .image_panel import ImagePanel from .image_panel import ImagePanel
_VIEW_ROTATE_SENSITIVITY = 0.35 _VIEW_ROTATE_SENSITIVITY = 0.35
@@ -2,7 +2,7 @@
from __future__ import annotations from __future__ import annotations
from PyQt5.QtCore import QSize, Qt, pyqtSignal from PyQt5.QtCore import QSize, pyqtSignal
from PyQt5.QtWidgets import ( from PyQt5.QtWidgets import (
QCheckBox, QCheckBox,
QComboBox, QComboBox,
@@ -14,11 +14,11 @@ from PyQt5.QtWidgets import (
QVBoxLayout, QVBoxLayout,
QWidget, QWidget,
) )
from orisys.finger import DEFAULT_FINGER_EXPORT_DIR
from viz.render_2d import ArrowBackground from viz.render_2d import ArrowBackground
from ..layout import DashboardLayout, LAYOUT from orisys.finger import DEFAULT_FINGER_EXPORT_DIR
from ..layout import LAYOUT, DashboardLayout
from ..mesh_view_state import MeshViewState from ..mesh_view_state import MeshViewState
from ..transport_icons import reset_icon, transport_icon from ..transport_icons import reset_icon, transport_icon
+16 -10
View File
@@ -7,15 +7,18 @@ ORY-MINI:单传感器基础流程
3. 按需计算接触区域 3. 按需计算接触区域
4. 读取并可视化结果 4. 读取并可视化结果
SDK version: 0.5.0
""" """
import os import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2
import orisys
import numpy as np
import argparse import argparse
import cv2
import numpy as np
import orisys
def main(): def main():
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
parser.add_argument("--video","-v", type=str, default="0", help="视频源:摄像头编号或视频文件路径") parser.add_argument("--video","-v", type=str, default="0", help="视频源:摄像头编号或视频文件路径")
@@ -30,11 +33,13 @@ def main():
# 输入既可以是摄像头编号,也可以是视频文件路径 # 输入既可以是摄像头编号,也可以是视频文件路径
try: try:
input = int(args.video) input = int(args.video)
is_camera = True except ValueError:
sensor = orisys.Sensor(input, config_name=args.config, verbose=args.verbose) # Not a camera index: treat the argument as a video-file path. Falling back
except: # only on the conversion (not on Sensor construction) keeps a real
is_camera = False # constructor failure visible instead of retrying it as a file path.
sensor = orisys.Sensor(args.video, config_name=args.config, verbose=args.verbose) sensor = orisys.Sensor(args.video, config_name=args.config, verbose=args.verbose)
else:
sensor = orisys.Sensor(input, config_name=args.config, verbose=args.verbose)
print("\n按键说明:'q' 退出程序,'r' 重置追踪器。") print("\n按键说明:'q' 退出程序,'r' 重置追踪器。")
@@ -90,9 +95,10 @@ def main():
cv2.imshow("depth", cv2.applyColorMap(div_normalized, cv2.COLORMAP_JET)) cv2.imshow("depth", cv2.applyColorMap(div_normalized, cv2.COLORMAP_JET))
# 步骤 3.4:打印关键结果 # 步骤 3.4:打印关键结果
fn_unit, fx_unit, fy_unit = sensor.force_units
print( print(
f"FPS={fps:.2f}, 法向力={fnormal:.4f}, 切向力X={fshearx:.4f}, " f"FPS={fps:.2f}, 法向力={fnormal:.4f} {fn_unit}, 切向力X={fshearx:.4f} {fx_unit}, "
f"切向力Y={fsheary:.4f}, 深度图尺寸={depth_map.shape}, 原始分辨率={sensor.img_size_raw}" f"切向力Y={fsheary:.4f} {fy_unit}, 深度图尺寸={depth_map.shape}, 原始分辨率={sensor.img_size_raw}"
) )
# 步骤 3.5:键盘控制 # 步骤 3.5:键盘控制
+308
View File
@@ -0,0 +1,308 @@
#!/usr/bin/env python3
"""Multi-sensor real-time example: N streams, one process each.
This is the deployment shape the multi-sensor throughput numbers assume. Each
stream owns a process and pipelines the per-frame work into three concurrent
stages (capture+unroll -> DIS flow -> HHD+readout), so a stream's rate is bounded
by its slowest stage instead of the sum of all three.
# five finger streams from a video file (benchmark / smoke test)
python examples/multi_stream.py --streams 5 -c finger -v data/dd02-0703/1.mp4
# five live cameras, camera ids 0..4
python examples/multi_stream.py --streams 5 -c finger --camera-base 0
# single box sensor
python examples/multi_stream.py --streams 1 -c box -v data/long.mp4
# five streams, additionally recording an arrow-overlay video per stream
# plus one 5-up mosaic, without stalling the pipeline
python examples/multi_stream.py --streams 5 -c finger -v data/dd02-0703/1.mp4 \
--seconds 10 --record outputs/arrows --mosaic
Notes
-----
* One process per stream. Threads inside a single process share the GIL and,
measured on a 9950X, five finger streams reach ~150 FPS aggregate as threads
versus ~626 FPS as processes.
* ``cv2.setNumThreads(2)`` per process: at five streams that beats 1, 4 and 8.
* Run one stream per process even when the sensor count is small; the process
boundary also isolates a crashed camera from the rest.
* ``--record`` draws the DIS flow with ``orisys.util.draw_arrows`` (plus the
contact overlay with ``--contact``) and encodes it with ``cv2.VideoWriter``.
Both happen on a dedicated thread fed by a bounded drop-oldest queue, so the
pipeline never waits on the encoder; frames the writer cannot keep up with are
counted in the ``rec`` column as ``written/dropped``. Keep ``--record`` for
verification runs - the encoder still competes for CPU with the pipeline.
"""
from __future__ import annotations
import argparse
import json
import multiprocessing as mp
import os
import statistics as st
import sys
import time
os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0")
import orisys
from orisys import StreamPipeline, StreamRecorder, configs
def _make_stream(args, index):
"""Return (sensor, produce, close) for one stream."""
source = args.video
if args.camera_base is not None:
source = int(args.camera_base) + index
if source is None:
raise ValueError("pass --video PATH or --camera-base ID")
cfg = configs.load_config(args.config)
if str(cfg.get("type", "")).lower() == "single_camera":
from orisys.finger import FingerRuntime
runtime = FingerRuntime.open(config=args.config, video=source,
verbose=False)
return runtime.tactile.raw_sensor, _finger_produce(runtime), runtime.close
sensor = orisys.Sensor(source, config_name=args.config, isstitch=True,
cuda=False, backend="cpu", verbose=False)
def produce():
return sensor.get_img()
return sensor, produce, sensor.disconnect
def _finger_produce(runtime):
def produce():
ok, frame = runtime.grab_and_process(color=False)
return frame if ok else None
return produce
def worker(args, index, out_path):
"""One stream: own process, three-stage pipeline."""
import cv2
cv2.setNumThreads(int(args.threads))
sensor, produce, close = _make_stream(args, index)
frames = []
stamps = []
started = time.perf_counter()
def collect(snap):
frames.append(snap.age_ms)
stamps.append(time.perf_counter())
# Recording is entirely off the hot path: ``on_frame`` only pushes onto a
# bounded drop-oldest queue, and a separate thread does the drawing and the
# H.264/MPEG-4 encoding. The pipeline never blocks on the encoder.
recorder = None
on_frame = collect
if args.record:
recorder = StreamRecorder(
os.path.join(args.record, f"stream_{index}.mp4"),
fps=args.record_fps,
every=args.record_every,
fourcc=args.record_fourcc,
valid_mask=lambda: getattr(sensor, "valid_mask", None),
grid_spacing=args.record_grid,
threshold=args.record_threshold,
arrow_scale=args.record_arrow_scale,
draw_contact=args.contact,
label=f"#{index}",
verbose=True,
)
def on_frame(snap, _collect=collect, _rec=recorder):
_collect(snap)
_rec.on_frame(snap)
if args.no_arrays:
include = ()
elif recorder is not None:
include = ("image", "flow") # what the arrow overlay needs
else:
include = ("image",)
pipe = StreamPipeline(sensor, produce, on_frame=on_frame,
include=include, contact=args.contact, depth=args.depth)
pipe.start()
record_stats = None
try:
deadline = started + args.seconds if args.seconds else None
while True:
if not pipe.running:
break
if args.frames and pipe.stats.published >= args.frames:
break
if deadline is not None and time.perf_counter() >= deadline:
break
time.sleep(0.002)
finally:
try:
pipe.stop()
except BaseException as exc: # surface a stage failure
print(f"[stream {index}] pipeline error: {type(exc).__name__}: {exc}",
file=sys.stderr)
raise
finally:
if recorder is not None:
record_stats = recorder.close()
close()
wall = time.perf_counter() - started
# Steady-state rate: skip the first frames, which pay the JIT / HHD-cache /
# DIS-reference-frame warm-up and would drag a short run's average down.
warm = min(30, max(0, len(stamps) - 2))
span = (stamps[-1] - stamps[warm]) if len(stamps) > warm + 1 else 0.0
steady = (len(stamps) - 1 - warm) / span if span > 0 else 0.0
payload = {
"index": index,
"published": pipe.stats.published,
"wall": wall,
"fps": pipe.stats.published / wall if wall else 0.0,
"steady_fps": steady,
"stats": pipe.stats.as_dict(),
"age_p95_ms": float(st.quantiles(frames, n=20)[18]) if len(frames) >= 20 else 0.0,
"record": record_stats,
}
with open(out_path, "w") as fh:
json.dump(payload, fh)
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--streams", "-n", type=int, default=5)
ap.add_argument("--config", "-c", default="finger")
ap.add_argument("--video", "-v", default=None, help="video file shared by all streams")
ap.add_argument("--camera-base", type=int, default=None,
help="first camera id; stream i uses base + i")
ap.add_argument("--frames", type=int, default=0, help="stop after N frames (0 = until EOF)")
ap.add_argument("--seconds", type=float, default=10.0, help="max wall time per stream")
ap.add_argument("--threads", type=int, default=2, help="cv2.setNumThreads per stream")
ap.add_argument("--depth", type=int, default=2, help="queue depth between stages")
ap.add_argument("--contact", action="store_true", help="also run compute_contact")
ap.add_argument("--no-arrays", action="store_true",
help="scalars only (fastest; skips the snapshot image copy)")
ap.add_argument("--out-dir", default=None)
ap.add_argument("--record", default=None, metavar="DIR",
help="write one arrow-overlay mp4 per stream into DIR "
"(drawing+encoding run on a separate thread, off the "
"pipeline's critical path)")
ap.add_argument("--record-fps", type=float, default=30.0,
help="playback fps of the recorded files")
ap.add_argument("--record-every", type=int, default=1,
help="keep one frame in N (raise it if the writer falls behind)")
ap.add_argument("--record-fourcc", default="mp4v",
help="mp4v (built in) or MJPG/avi for maximum compatibility")
ap.add_argument("--record-grid", type=int, default=14, help="arrow grid spacing in px")
ap.add_argument("--record-threshold", type=float, default=0.1,
help="displacement (px) below which a dot is drawn instead of an arrow")
ap.add_argument("--record-arrow-scale", type=float, default=0.15,
help="draw_arrows divides the displacement by this; 0.15 magnifies "
"the finger's ~1-3 px field ~6.7x so directions are legible")
ap.add_argument("--mosaic", action="store_true",
help="after the run, tile the recordings into one grid video")
ap.add_argument("--mosaic-cols", type=int, default=0, help="mosaic columns (0 = square-ish)")
ap.add_argument("--mosaic-scale", type=float, default=1.0, help="mosaic downscale factor")
args = ap.parse_args()
if args.record and args.no_arrays:
ap.error("--record needs the frame and flow arrays; drop --no-arrays")
if args.mosaic and not args.record:
ap.error("--mosaic needs --record DIR")
if args.record:
args.record = os.path.abspath(args.record)
os.makedirs(args.record, exist_ok=True)
out_dir = args.out_dir or os.path.join(os.path.dirname(os.path.abspath(__file__)),
"..", "outputs", "multi_stream")
out_dir = os.path.abspath(out_dir)
os.makedirs(out_dir, exist_ok=True)
ctx = mp.get_context("spawn")
procs, paths = [], []
for i in range(args.streams):
path = os.path.join(out_dir, f"stream_{i}.json")
paths.append(path)
procs.append(ctx.Process(target=worker, args=(args, i, path)))
for p in procs:
p.start()
for p in procs:
p.join()
rows = []
for path in paths:
try:
with open(path) as fh:
rows.append(json.load(fh))
except Exception: # noqa: S110 - a stream that wrote no result is reported below
# A stream that crashed or produced no JSON output is expected here;
# the empty-``rows`` check below reports that case explicitly, so this
# is not a hidden failure.
pass
if not rows:
print("no stream produced results", file=sys.stderr)
return 1
wall = max(r["wall"] for r in rows)
total = sum(r["published"] for r in rows)
steady_rows = [r for r in rows if r.get("steady_fps")]
steady = st.mean(r["steady_fps"] for r in steady_rows) if steady_rows else 0.0
print(f"\nstreams={len(rows)} frames={total} wall={wall:.2f}s")
print(f"aggregate throughput : {total / wall:.1f} FPS (incl. warm-up)")
print(f"per-stream throughput : {total / wall / len(rows):.1f} FPS (incl. warm-up)")
print(f"per-stream steady-state : {steady:.1f} FPS")
recording = args.record is not None
print(f"\n{'stream':>7}{'published':>11}{'FPS':>9}{'steady':>9}{'produce':>10}"
f"{'flow':>9}{'hhd':>9}{'age':>9}{'age p95':>10}"
+ (f"{'rec':>13}" if recording else ""))
for r in sorted(rows, key=lambda x: x["index"]):
s = r["stats"]
rec = ""
if recording:
rs = r.get("record") or {}
rec = f"{rs.get('written', 0)}/{rs.get('dropped', 0)}"
if rs.get("error"):
rec = "ERR"
print(f"{r['index']:>7}{r['published']:>11}{r['fps']:>9.1f}"
f"{r.get('steady_fps', 0.0):>9.1f}"
f"{s['produce_ms']:>10.2f}{s['flow_ms']:>9.2f}{s['hhd_ms']:>9.2f}"
f"{s['age_ms']:>9.2f}{r['age_p95_ms']:>10.2f}"
+ (f"{rec:>13}" if recording else ""))
if recording:
print("rec column = frames written / frames dropped by the writer thread")
for r in sorted(rows, key=lambda x: x["index"]):
rs = r.get("record") or {}
if rs.get("error"):
print(f" stream {r['index']}: {rs['error']}")
if args.mosaic and recording:
streams = sorted(r["index"] for r in rows)
srcs = [os.path.join(args.record, f"stream_{i}.mp4") for i in streams]
missing = [p for p in srcs if not os.path.isfile(p)]
if missing:
print(f"mosaic skipped, missing {missing}", file=sys.stderr)
else:
out = os.path.join(args.record, "mosaic.mp4")
# No tile labels: each recorded frame already carries the "#N" HUD.
info = orisys.mosaic_videos(srcs, out, cols=args.mosaic_cols or None,
scale=args.mosaic_scale,
fps=args.record_fps)
print(f"\nmosaic -> {info['path']} {info['size'][0]}x{info['size'][1]} "
f"{info['grid'][0]}x{info['grid'][1]} grid {info['frames']} frames "
f"@ {info['fps']:g} fps")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+3 -2
View File
@@ -4,9 +4,10 @@ from __future__ import annotations
from dataclasses import dataclass from dataclasses import dataclass
from orisys.finger import DEFAULT_FINGER_EXPORT_DIR
from viz.mesh_camera import DEFAULT_FINGER_VIZ_POSE from viz.mesh_camera import DEFAULT_FINGER_VIZ_POSE
from orisys.finger import DEFAULT_FINGER_EXPORT_DIR
def mesh_threshold(mesh_map: str, threshold: float | None) -> float: def mesh_threshold(mesh_map: str, threshold: float | None) -> float:
if threshold is not None: if threshold is not None:
@@ -33,7 +34,7 @@ class PipelineConfig:
profile_every: int = 30 profile_every: int = 30
@classmethod @classmethod
def from_args(cls, args) -> "PipelineConfig": def from_args(cls, args) -> PipelineConfig:
return cls( return cls(
video=args.video, video=args.video,
config=args.config, config=args.config,
-1
View File
@@ -7,7 +7,6 @@ from dataclasses import dataclass
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
CSV_COLUMNS = ( CSV_COLUMNS = (
"frame_index", "frame_index",
"timestamp_s", "timestamp_s",
+2 -2
View File
@@ -6,10 +6,10 @@ import time
from dataclasses import dataclass from dataclasses import dataclass
import numpy as np import numpy as np
from orisys.finger import FingerRuntime, compute_uv_surface_frames
from viz import FrameRecorder, build_uv_surface_mesh, resolve_viz_pose_path from viz import FrameRecorder, build_uv_surface_mesh, resolve_viz_pose_path
from orisys.finger import FingerRuntime, compute_uv_surface_frames
from .config import PipelineConfig from .config import PipelineConfig
from .csv_recorder import CsvMeasurement from .csv_recorder import CsvMeasurement
from .frame import FingerFrameView from .frame import FingerFrameView
+7 -3
View File
@@ -16,10 +16,10 @@ os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2 import cv2
import numpy as np import numpy as np
import orisys
from viz import ForceChartWindow, FrameRecorder from viz import ForceChartWindow, FrameRecorder
import orisys
def open_sensor(video: str, config: str, cal: str, verbose: bool): def open_sensor(video: str, config: str, cal: str, verbose: bool):
try: try:
@@ -65,7 +65,11 @@ def main():
sensor.info.VRAW, sensor.info.VRAW,
sensor.info.DEPTH, sensor.info.DEPTH,
) )
print(f"FPS={fps:.2f}, 法向力={fn:.4f}, 切向力X={fx:.4f}, 切向力Y={fy:.4f}") fn_unit, fx_unit, fy_unit = sensor.force_units
print(
f"FPS={fps:.2f}, 法向力={fn:.4f} {fn_unit}, "
f"切向力X={fx:.4f} {fx_unit}, 切向力Y={fy:.4f} {fy_unit}"
)
arrows = orisys.util.draw_arrows(sensor.img, flow, threshold=5, grid_spacing=10, arrow_scale=1.0) arrows = orisys.util.draw_arrows(sensor.img, flow, threshold=5, grid_spacing=10, arrow_scale=1.0)
cv2.imshow("形变矢量场", arrows) cv2.imshow("形变矢量场", arrows)
+23 -12
View File
@@ -18,17 +18,19 @@
- Windows 下必须在 if __name__ == '__main__' 中调用 mp.freeze_support() - Windows 下必须在 if __name__ == '__main__' 中调用 mp.freeze_support()
- 每个传感器进程都会占用独立的 GPU 显存 - 每个传感器进程都会占用独立的 GPU 显存
SDK version: 0.5.0
""" """
import os import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2
import orisys
import numpy as np
import multiprocessing as mp
from multiprocessing import Queue, Process
import time
import argparse import argparse
import multiprocessing as mp
import queue
import time
from multiprocessing import Process, Queue
import cv2
import orisys
def sensor_worker(sensor_id, vid_src, config_name, result_queue, stop_event, target_fps=None): def sensor_worker(sensor_id, vid_src, config_name, result_queue, stop_event, target_fps=None):
@@ -56,6 +58,7 @@ def sensor_worker(sensor_id, vid_src, config_name, result_queue, stop_event, tar
print(f"传感器 {sensor_id} 已启动" + (f"(目标帧率:{target_fps} FPS)" if target_fps else "")) print(f"传感器 {sensor_id} 已启动" + (f"(目标帧率:{target_fps} FPS)" if target_fps else ""))
frame_interval = 1.0 / target_fps if target_fps and target_fps > 0 else 0.0 frame_interval = 1.0 / target_fps if target_fps and target_fps > 0 else 0.0
dropped_frames = 0
while not stop_event.is_set(): while not stop_event.is_set():
t_start = time.perf_counter() t_start = time.perf_counter()
@@ -96,9 +99,10 @@ def sensor_worker(sensor_id, vid_src, config_name, result_queue, stop_event, tar
# 步骤 6:将结果发送到主进程(非阻塞) # 步骤 6:将结果发送到主进程(非阻塞)
try: try:
result_queue.put_nowait(data) result_queue.put_nowait(data)
except Exception: except queue.Full:
# 如果队列已满,则丢弃当前帧,避免阻塞工作进程 # 队列已满:丢弃当前帧,避免阻塞工作进程。这是有意的背压策略,
pass # 但丢帧必须可观测,否则主进程看到的低帧率无从解释。
dropped_frames += 1
except Exception as e: except Exception as e:
print(f"传感器 {sensor_id} 处理失败:{e}") print(f"传感器 {sensor_id} 处理失败:{e}")
@@ -113,7 +117,13 @@ def sensor_worker(sensor_id, vid_src, config_name, result_queue, stop_event, tar
# 释放资源 # 释放资源
sensor.disconnect() sensor.disconnect()
print(f"传感器 {sensor_id} 已停止。") if dropped_frames:
print(
f"传感器 {sensor_id} 已停止(因结果队列已满丢弃 {dropped_frames} 帧,"
"主进程处理慢于采集)。"
)
else:
print(f"传感器 {sensor_id} 已停止。")
except Exception as e: except Exception as e:
print(f"传感器 {sensor_id} 初始化失败:{e}") print(f"传感器 {sensor_id} 初始化失败:{e}")
@@ -194,7 +204,8 @@ def main():
while True: while True:
try: try:
data = queue.get_nowait() data = queue.get_nowait()
except: except queue.Empty:
# 队列已排空:正常结束内层循环,不是错误。
break break
if data is not None: if data is not None:
+9 -9
View File
@@ -1,5 +1,7 @@
"""Shared visualization helpers for Orisys example scripts.""" """Shared visualization helpers for Orisys example scripts."""
from .finger_mesh_viewer import FingerMeshViewer
from .flow_arrow import DEFAULT_FLOW_ARROW_STYLE, FlowArrowOverlay, FlowArrowStyle
from .force_charts import ForceChartWindow from .force_charts import ForceChartWindow
from .mesh_camera import ( from .mesh_camera import (
DEFAULT_FINGER_VIZ_POSE, DEFAULT_FINGER_VIZ_POSE,
@@ -8,33 +10,32 @@ from .mesh_camera import (
camera_look_at_from_world_to_cam, camera_look_at_from_world_to_cam,
resolve_viz_pose_path, resolve_viz_pose_path,
) )
from .mesh_viewer import UVMeshViewer, build_uv_surface_mesh
from .offscreen_mesh import OffscreenFingerMeshRenderer from .offscreen_mesh import OffscreenFingerMeshRenderer
from .recording import FrameRecorder
from .render_2d import ( from .render_2d import (
DEFAULT_ARROW_CANVAS_COLORS,
ArrowBackground, ArrowBackground,
ArrowCanvasColors, ArrowCanvasColors,
DEFAULT_ARROW_CANVAS_COLORS,
depth_colormap, depth_colormap,
frame_to_bgr, frame_to_bgr,
render_contact_panel, render_contact_panel,
render_depth_panel, render_depth_panel,
render_uv_arrows, render_uv_arrows,
) )
from .flow_arrow import DEFAULT_FLOW_ARROW_STYLE, FlowArrowOverlay, FlowArrowStyle
from .finger_mesh_viewer import FingerMeshViewer
from .mesh_viewer import UVMeshViewer, build_uv_surface_mesh
from .recording import FrameRecorder
from .signal_filters import lowpass_filter from .signal_filters import lowpass_filter
__all__ = [ __all__ = [
"ArrowBackground",
"ArrowCanvasColors",
"DEFAULT_ARROW_CANVAS_COLORS", "DEFAULT_ARROW_CANVAS_COLORS",
"DEFAULT_FLOW_ARROW_STYLE", "DEFAULT_FLOW_ARROW_STYLE",
"ArrowBackground",
"ArrowCanvasColors",
"FingerMeshViewer",
"FlowArrowOverlay", "FlowArrowOverlay",
"FlowArrowStyle", "FlowArrowStyle",
"FingerMeshViewer",
"ForceChartWindow", "ForceChartWindow",
"FrameRecorder", "FrameRecorder",
"OffscreenFingerMeshRenderer",
"UVMeshViewer", "UVMeshViewer",
"apply_mesh_lighting", "apply_mesh_lighting",
"apply_view_from_world_to_cam", "apply_view_from_world_to_cam",
@@ -43,7 +44,6 @@ __all__ = [
"depth_colormap", "depth_colormap",
"frame_to_bgr", "frame_to_bgr",
"lowpass_filter", "lowpass_filter",
"OffscreenFingerMeshRenderer",
"render_contact_panel", "render_contact_panel",
"render_depth_panel", "render_depth_panel",
"render_uv_arrows", "render_uv_arrows",
+5 -1
View File
@@ -8,7 +8,11 @@ import numpy as np
from orisys.finger.viewer import UVMeshViewer from orisys.finger.viewer import UVMeshViewer
from .contact_arrow import ContactArrowOverlay, ContactArrowStyle, DEFAULT_CONTACT_ARROW_STYLE from .contact_arrow import (
DEFAULT_CONTACT_ARROW_STYLE,
ContactArrowOverlay,
ContactArrowStyle,
)
from .flow_arrow import DEFAULT_FLOW_ARROW_STYLE, FlowArrowOverlay, FlowArrowStyle from .flow_arrow import DEFAULT_FLOW_ARROW_STYLE, FlowArrowOverlay, FlowArrowStyle
+2 -1
View File
@@ -2,8 +2,9 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass from dataclasses import dataclass
from typing import Callable, Literal from typing import Literal
import cv2 import cv2
import numpy as np import numpy as np
+5 -1
View File
@@ -1,4 +1,8 @@
"""Re-exports for finger 3D mesh examples; core viewer lives in orisys.finger.""" """Re-exports for finger 3D mesh examples; core viewer lives in orisys.finger."""
from orisys.finger.mesh import * # noqa: F403 from orisys.finger.mesh import * # noqa: F403
from orisys.finger.viewer import UVMeshViewer, init_uv_mesh_viewer, update_uv_mesh_viewer # noqa: F401 from orisys.finger.viewer import ( # noqa: F401
UVMeshViewer,
init_uv_mesh_viewer,
update_uv_mesh_viewer,
)
-1
View File
@@ -2,7 +2,6 @@
from __future__ import annotations from __future__ import annotations
import os
from pathlib import Path from pathlib import Path
import cv2 import cv2
+1
View File
@@ -7,6 +7,7 @@ from typing import TYPE_CHECKING, Literal
import cv2 import cv2
import numpy as np import numpy as np
import orisys import orisys
if TYPE_CHECKING: if TYPE_CHECKING:
+1
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import numpy as np import numpy as np
from scipy.signal import butter, filtfilt from scipy.signal import butter, filtfilt
def lowpass_filter(data, cutoff_hz=5, sample_rate_hz=30, order=2) -> np.ndarray: def lowpass_filter(data, cutoff_hz=5, sample_rate_hz=30, order=2) -> np.ndarray:
"""Zero-phase Butterworth low-pass filter for uniformly sampled 1-D data.""" """Zero-phase Butterworth low-pass filter for uniformly sampled 1-D data."""
y = np.asarray(data, dtype=np.float64) y = np.asarray(data, dtype=np.float64)
+8 -5
View File
@@ -99,9 +99,12 @@ target = "examples"
patterns = ["*.py", "README.md", "viz/**/*.py", "pipeline/**/*.py", "gui/**/*.py"] patterns = ["*.py", "README.md", "viz/**/*.py", "pipeline/**/*.py", "gui/**/*.py"]
preserve_tree = true preserve_tree = true
[[tool.orisys.bundle.items]] # NOTE: docs/ is deliberately NOT bundled into the wheel.
name = "docs" # Documentation ships with the GitHub repository / PyPI project page only:
source = "docs" # - docs/publishing.md is an internal process document
target = "docs" # - docs/pypi_readme.md is read from the checkout at build time as the PyPI
patterns = ["*.md", "*.pdf"] # long description (build/wheel/package.py::PYPI_README_RELPATH), never shipped
# - keeping docs out means editing a doc no longer invalidates built wheels
# Should an offline API reference be needed later, add a narrower item that ships
# only docs/api.md.
+4
View File
@@ -1,5 +1,9 @@
[pytest] [pytest]
testpaths = tests testpaths = tests
python_files = test_*.py python_files = test_*.py
# Run against the checkout without requiring `pip install -e .`, so a stale
# orisys in site-packages cannot silently shadow src/orisys.
pythonpath = src
markers = markers =
slow: long-running tests for release validation slow: long-running tests for release validation
hardware: requires a physical sensor/camera attached to the machine
+55
View File
@@ -0,0 +1,55 @@
# Ruff configuration — pinned deliberately.
#
# Before this file existed the repo had no Ruff config, so `ruff check .` ran whatever the
# installed Ruff version enables by default (Ruff 0.16 turns on a very broad set: UP/RUF/PLR/
# S/TRY/PERF/FURB/YTT/ASYNC...). The same tree therefore produced different findings on
# different machines and between a developer and a runner. Never rely on tool defaults.
#
# Policy, decided from the measured findings rather than from taste:
#
# * `select` is explicit. High-signal correctness rules first, then only the mechanical
# modernisation rules we are willing to apply without human review.
# * BLE001 is selected so existing `# noqa: BLE001 - <reason>` suppressions stay *used* and
# RUF100 can never delete their rationale — a broad `except Exception` is a legitimate
# choice around device I/O, capture probes and teardown. It is NOT part of the gate.
# * S110 / S112 / E722 ARE part of the gate: a silent handler hides a failure. This matters
# most in the live capture path, where a swallowed read error kills a source silently.
# * RUF100 reports unused suppressions but is `unfixable`: a machine must not remove a
# suppression whose comment records why it exists. Triage those by hand.
# * The mechanical sweep never uses `--unsafe-fixes`.
#
# The gate selects the narrow set explicitly, so BLE001 cannot fail a build, and only ever
# touches changed files:
#
# tools/lint_gate.py # the gate: changed files, machine-readable
target-version = "py310"
line-length = 88
[lint]
select = [
"E9", "E722", # syntax/IO errors, and bare `except:`
"E402", # module import not at top of file — SELECTED for visibility, NOT gated.
# `import` after `os.environ[...]` is deliberate in examples/ and scripts/
# (the environment must be set before Qt/cv2/orisys load). E402 is the only
# rule that would catch an accidental import moved above that setup, so it
# stays reported, without failing a build. An intentional site belongs in a
# `# noqa: E402 - <reason>` when it is touched.
"F", # pyflakes: unused import, redefinition, undefined name
"S110", "S112", # try-except-pass / try-except-continue — an invisible failure
"B023", # function definition does not bind loop variable
"PLW1510", # subprocess.run without an explicit check
"RUF100", # unused noqa — reported for triage, never auto-removed
"UP", # pyupgrade
"I", # import sorting
"RUF022", "RUF023", # __all__ / __slots__ ordering
"PIE", # flake8-pie
"RET501", "PLR1711",
"SIM114",
"BLE001", # selected for suppression hygiene only — excluded from the gate
]
unfixable = ["RUF100"]
[format]
quote-style = "double"
indent-style = "space"
+11 -7
View File
@@ -1,6 +1,7 @@
import os import os
import sys import sys
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2 import cv2
@@ -9,10 +10,11 @@ _local_src = os.path.join(os.path.dirname(__file__), "src")
if _local_src not in sys.path: if _local_src not in sys.path:
sys.path.insert(0, _local_src) sys.path.insert(0, _local_src)
import orisys
import numpy as np
import time
import argparse import argparse
import time
import orisys
def main(): def main():
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
@@ -30,11 +32,13 @@ def main():
# 方式1: 从视频文件读取 # 方式1: 从视频文件读取
try: try:
input = int(args.video) input = int(args.video)
is_camera = True except ValueError:
sensor = orisys.Sensor(input, config_name=args.config, cal_path= args.cal, verbose=args.verbose, debug=args.debug) # Not a camera index: treat the argument as a video-file path. Only the
except: # conversion is guarded, so a genuine Sensor construction error is not
is_camera = False # swallowed and retried as a file path.
sensor = orisys.Sensor(args.video, config_name=args.config, cal_path= args.cal, verbose=args.verbose, debug=args.debug) sensor = orisys.Sensor(args.video, config_name=args.config, cal_path= args.cal, verbose=args.verbose, debug=args.debug)
else:
sensor = orisys.Sensor(input, config_name=args.config, cal_path= args.cal, verbose=args.verbose, debug=args.debug)
# sensor = orisys.Sensor(2, config_name="./config/ddjx01_q6a.json", cal_path= "./config/ddjx01.npy", isstitch=True, cuda=False, verbose=True, backend='cpu') # sensor = orisys.Sensor(2, config_name="./config/ddjx01_q6a.json", cal_path= "./config/ddjx01.npy", isstitch=True, cuda=False, verbose=True, backend='cpu')
# 方式2: 从摄像头读取(取消注释使用) # 方式2: 从摄像头读取(取消注释使用)
+7 -4
View File
@@ -1,12 +1,15 @@
import os import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2
import orisys
import numpy as np
import time import time
import cv2
import numpy as np
import pyqtgraph as pg import pyqtgraph as pg
import orisys
def create_plot(): def create_plot():
p1 = win.addPlot(title="法相力") p1 = win.addPlot(title="法相力")
p1.setLabel('bottom', '时间', units='s') p1.setLabel('bottom', '时间', units='s')
@@ -149,7 +152,7 @@ def main():
print("\n 停止程序") print("\n 停止程序")
break break
if key == ord("s"): if key == ord("s"):
save_path = f"./data/pulse_test.mp4" save_path = "./data/pulse_test.mp4"
# Ensure directory exists # Ensure directory exists
os.makedirs(os.path.dirname(save_path), exist_ok=True) os.makedirs(os.path.dirname(save_path), exist_ok=True)
# Get frame dimensions from the actual frame # Get frame dimensions from the actual frame
+16 -9
View File
@@ -46,7 +46,7 @@ import os
import shlex import shlex
import sys import sys
from pathlib import Path from pathlib import Path
from typing import Optional, Union from typing import Union
os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0") os.environ.setdefault("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS", "0")
@@ -60,13 +60,14 @@ except ImportError as exc: # pragma: no cover
) from exc ) from exc
import cv2 import cv2
from orisys.capture import is_live_camera, list_cameras from orisys.capture import is_live_camera, list_cameras
from orisys.finger import FingerRuntime from orisys.finger import FingerRuntime
Source = Union[int, str] Source = Union[int, str]
def _parse_source(token: str) -> Optional[Source]: def _parse_source(token: str) -> Source | None:
"""Accept camera index int, numeric string, V4L path, or filesystem path.""" """Accept camera index int, numeric string, V4L path, or filesystem path."""
text = str(token).strip().strip('"').strip("'") text = str(token).strip().strip('"').strip("'")
if not text: if not text:
@@ -116,9 +117,9 @@ class UartFingerController:
self.show = show self.show = show
self.verbose = verbose self.verbose = verbose
self.runtime: Optional[FingerRuntime] = None self.runtime: FingerRuntime | None = None
self._connected_source: Optional[Source] = None self._connected_source: Source | None = None
self._last_search_source: Optional[Source] = None self._last_search_source: Source | None = None
self._streaming = False self._streaming = False
self._should_quit = False self._should_quit = False
@@ -134,14 +135,18 @@ class UartFingerController:
if self.runtime is not None: if self.runtime is not None:
try: try:
self.runtime.close() self.runtime.close()
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001, S110 - teardown must not raise
# Best-effort release: a close failure here has no recovery path in
# a shutdown request handler, and raising would skip the reset below.
pass pass
self.runtime = None self.runtime = None
self._connected_source = None self._connected_source = None
if self.show: if self.show:
try: try:
cv2.destroyAllWindows() cv2.destroyAllWindows()
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001, S110 - no GUI window to destroy
# Expected when no window was ever created (headless/--no-show run):
# this is a normal no-op, not a hidden failure.
pass pass
def _resolve_connect_source(self, args: list[str]) -> Source: def _resolve_connect_source(self, args: list[str]) -> Source:
@@ -220,7 +225,7 @@ class UartFingerController:
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001
return False return False
def _poll_command_line(self, *, blocking: bool = True) -> Optional[str]: def _poll_command_line(self, *, blocking: bool = True) -> str | None:
"""Read one command line if available; return None on timeout/empty.""" """Read one command line if available; return None on timeout/empty."""
if not blocking and self.ser.in_waiting <= 0: if not blocking and self.ser.in_waiting <= 0:
return None return None
@@ -447,7 +452,9 @@ def main(argv: list[str] | None = None) -> int:
print("\nInterrupted.", flush=True) print("\nInterrupted.", flush=True)
try: try:
controller.reply("OK quit") controller.reply("OK quit")
except Exception: # noqa: BLE001 except Exception: # noqa: BLE001, S110 - peer may already be gone
# We are already shutting down on Ctrl-C; failing to send the final
# acknowledgement (peer closed the port) must not mask the exit path.
pass pass
finally: finally:
controller.close_runtime() controller.close_runtime()
+340
View File
@@ -0,0 +1,340 @@
"""Measure normal-force zero-point drift, live or from a recording.
Phase 0 of the zero-point plan: quantify the drift before changing the pipeline.
The number this prints is what the acceptance thresholds get set from, so it is
deliberately explicit about which rows it used and what it could not conclude.
Usage
-----
Live capture (needs a sensor; writes a recording directory when --out is given):
python scripts/measure_zero_drift.py --live --seconds 1800 --out D:\\tmp\\idle
From an existing recording (the directory holding forces.csv):
python scripts/measure_zero_drift.py --recording <dir> --unit both
python scripts/measure_zero_drift.py --recording <dir> --warmup 5 --start 10 --stop 60
What it reports
---------------
Per axis, over the analysed window:
* mean / median / stdev / peak-to-peak, in raw counts and (with --unit N) newtons
* a least-squares slope in counts/s -- the ramp that "zero drift" actually means
* a drift ratio (end-third mean minus start-third mean), which is insensitive to
whether the drift is linear or a slow wander
* the band left after removing the window mean, i.e. what a perfect single-point
re-zero would achieve -- this separates "the zero moved" from "the noise is wide"
The raw columns are ``FN_raw, FX_raw, FY_raw`` and the calibrated ones are
``FN_N, FX_N, FY_N`` (see ``FORCES_HEADER`` in
``orisys_app_shared.record.experiment``); rows with ``valid != 1`` are excluded
rather than read as zeros.
"""
from __future__ import annotations
import argparse
import csv
import sys
from collections.abc import Sequence
from pathlib import Path
import numpy as np
RAW_AXES = ("FN_raw", "FX_raw", "FY_raw")
NEWTON_AXES = ("FN_N", "FX_N", "FY_N")
TIME_COLUMN = "t_rel_s"
VALID_COLUMN = "valid"
def _load_rows(source: Path) -> list[dict[str, str]]:
path = source / "forces.csv" if source.is_dir() else source
if not path.is_file():
raise SystemExit(f"no forces.csv at {path}")
with path.open(encoding="utf-8", newline="") as handle:
return list(csv.DictReader(handle))
def _column(rows: Sequence[dict[str, str]], name: str) -> np.ndarray:
values = []
for row in rows:
raw = row.get(name)
try:
values.append(float(raw)) # type: ignore[arg-type]
except (TypeError, ValueError):
values.append(np.nan)
return np.asarray(values, dtype=np.float64)
def _window(
rows: list[dict[str, str]],
*,
start: float | None,
stop: float | None,
warmup: float,
) -> tuple[np.ndarray, list[dict[str, str]]]:
times = _column(rows, TIME_COLUMN)
valid = _column(rows, VALID_COLUMN)
mask = np.ones(times.shape, dtype=bool)
if times.size:
mask &= ~np.isnan(times) # drop NaN timestamps (unparsable TIME cells)
if start is not None:
mask &= times >= float(start)
if stop is not None:
mask &= times <= float(stop)
# ``valid`` is authoritative: an invalid row is not a zero reading.
mask &= (valid == 1) | np.isnan(valid)
kept = [row for row, keep in zip(rows, mask) if keep]
if not kept:
return np.asarray([], dtype=np.float64), []
times = np.asarray([float(row[TIME_COLUMN]) for row in kept], dtype=np.float64)
if warmup > 0 and times.size:
t0 = times[0]
kept = [row for row in kept if float(row[TIME_COLUMN]) - t0 >= warmup]
times = np.asarray([float(row[TIME_COLUMN]) for row in kept], dtype=np.float64)
return times, kept
def _describe(label: str, values: np.ndarray, times: np.ndarray) -> dict[str, float]:
finite = values[np.isfinite(values)]
if finite.size == 0:
return {"axis": label, "n": 0}
mean = float(finite.mean())
median = float(np.median(finite))
# A min-to-max span does not shrink when a constant is removed, so it cannot
# answer "what would a re-zero achieve". The robust band and the inter-
# percentile width can, and they are what the noise is measured against in
# the existing perf analysis.
centred = finite - median
band_1sigma = float(
(np.percentile(centred, 84.1) - np.percentile(centred, 15.9)) / 2.0
)
width_99 = float(np.percentile(finite, 99.5) - np.percentile(finite, 0.5))
slope = float("nan")
if times.size == finite.size and times.size > 2 and float(times[-1] - times[0]) > 0:
slope = float(np.polyfit(times - times[0], finite, 1)[0])
third = max(1, finite.size // 3)
drift_ratio = float(finite[-third:].mean() - finite[:third].mean())
return {
"axis": label,
"n": int(finite.size),
"mean": mean,
"median": median,
"stdev": float(finite.std(ddof=0)),
"p2p": float(finite.max() - finite.min()),
"band_1sigma": band_1sigma,
"width_99": width_99,
"slope_per_s": slope,
"drift_ratio": drift_ratio,
}
def _print_table(title: str, stats: list[dict[str, float]], unit: str) -> None:
usable = [s for s in stats if s.get("n")]
print(f"\n=== {title} ===")
if not usable:
print(" no rows in the analysed window")
return
print(
f" {'axis':<8} {'n':>7} {'mean':>13} {'median':>13} {'stdev':>12} "
f"{'band1s':>11} {'w99':>11} {'slope/s':>11} {'drift':>11} [{unit}]"
)
for s in usable:
print(
f" {s['axis']:<8} {s['n']:>7} {s['mean']:>13.3f} {s['median']:>13.3f} "
f"{s['stdev']:>12.3f} {s['band_1sigma']:>11.3f} {s['width_99']:>11.3f} "
f"{s['slope_per_s']:>11.4f} {s['drift_ratio']:>11.3f}"
)
print(
" slope/s = least-squares ramp (the drift rate)\n"
" band1s = robust 1-sigma half-width about the median\n"
" w99 = 0.5-99.5 percentile width (tail-resistant spread)\n"
" drift = mean(last third) - mean(first third): moved, not just noisy"
)
def _analyse(rows: list[dict[str, str]], args: argparse.Namespace) -> int:
times, kept = _window(
rows, start=args.start, stop=args.stop, warmup=args.warmup
)
print(f"recording rows : {len(rows)}")
print(f"analysed rows : {len(kept)}")
if times.size:
dt = np.diff(times)
print(
f"analysed window : {times[0]:.3f} .. {times[-1]:.3f} s "
f"({times[-1] - times[0]:.3f} s)"
)
if dt.size and dt[dt > 0].size:
rate = 1.0 / float(np.median(dt[dt > 0]))
print(f"effective sample rate : {rate:.2f} Hz")
if len(kept) < 30:
print(
"\nNOTE: fewer than 30 analysed rows. This cannot characterise a slow\n"
" zero drift -- capture an idle run of at least several minutes."
)
if args.unit in ("raw", "both"):
_print_table(
"raw counts (pre-calibration)",
[_describe(axis, _column(kept, axis), times) for axis in RAW_AXES],
"counts",
)
if args.unit in ("N", "both"):
_print_table(
"newtons (what the operator sees)",
[_describe(axis, _column(kept, axis), times) for axis in NEWTON_AXES],
"N",
)
if args.out:
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
with out.open("w", encoding="utf-8", newline="") as handle:
writer = csv.writer(handle)
writer.writerow(
["axis", "unit", "n", "mean", "median", "stdev", "band_1sigma",
"width_99", "slope_per_s", "drift_ratio"]
)
groups = []
if args.unit in ("raw", "both"):
groups.append((RAW_AXES, "counts"))
if args.unit in ("N", "both"):
groups.append((NEWTON_AXES, "N"))
for axes, unit in groups:
for axis in axes:
s = _describe(axis, _column(kept, axis), times)
if not s.get("n"):
continue
writer.writerow([
s["axis"], unit, s["n"], f"{s['mean']:.6f}",
f"{s['median']:.6f}", f"{s['stdev']:.6f}",
f"{s['band_1sigma']:.6f}", f"{s['width_99']:.6f}",
f"{s['slope_per_s']:.6f}", f"{s['drift_ratio']:.6f}",
])
print(f"\nwrote {out}")
return 0
def _capture_live(args: argparse.Namespace) -> int:
"""Sample a real sensor with no load applied.
Uses only the public ``read_info``/``disconnect`` surface, so this keeps
working if the internals move.
"""
import time as _time
from orisys import Sensor
sensor = Sensor(
"external",
config_name=args.config,
verbose=args.verbose,
backend=args.backend,
)
seconds = float(args.seconds)
period = 1.0 / float(args.sample_rate)
rows: list[tuple[float, float, float, float, float, float, float]] = []
try:
print(
f"capturing {seconds:.0f}s at ~{args.sample_rate:.0f} Hz from config "
f"{args.config!r} ({args.backend}); keep the sensor unloaded"
)
t0 = _time.monotonic()
while True:
elapsed = _time.monotonic() - t0
if elapsed >= seconds:
break
raw = sensor.read_info(
sensor.info.FNORMAL_RAW,
sensor.info.FSHEARX_RAW,
sensor.info.FSHEARY_RAW,
)
newtons = sensor.read_info(
sensor.info.FNORMAL,
sensor.info.FSHEARX,
sensor.info.FSHEARY,
)
rows.append((elapsed, *[float(v) for v in raw], *[float(v) for v in newtons]))
_time.sleep(max(0.0, period - ((_time.monotonic() - t0) - elapsed)))
finally:
sensor.disconnect()
if not rows:
print("no samples captured (external-frame mode may need a frame source)")
return 1
times = np.asarray([row[0] for row in rows], dtype=np.float64)
print(f"\ncaptured {len(rows)} samples over {times[-1]:.1f}s")
if args.out:
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
path = out / "forces.csv"
with path.open("w", encoding="utf-8", newline="") as handle:
writer = csv.writer(handle)
writer.writerow(
["session_id", "frame_id", TIME_COLUMN, *RAW_AXES, *NEWTON_AXES,
VALID_COLUMN, "calibration_id"]
)
for index, row in enumerate(rows):
t_rel, *rest = row
writer.writerow(
[0, index, f"{t_rel:.6f}", *[f"{v:.6f}" for v in rest], 1, ""]
)
print(f"wrote {path}")
return 0
_print_table(
"live raw counts",
[
_describe(axis, np.asarray([row[1 + i] for row in rows]), times)
for i, axis in enumerate(RAW_AXES)
],
"counts",
)
_print_table(
"live newtons",
[
_describe(axis, np.asarray([row[4 + i] for row in rows]), times)
for i, axis in enumerate(NEWTON_AXES)
],
"N",
)
return 0
def main(argv: Sequence[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description="Measure normal-force zero-point drift (Phase 0)."
)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--recording", type=Path,
help="recording directory or forces.csv path")
source.add_argument("--live", action="store_true",
help="capture from a Sensor instead (external-frame mode)")
parser.add_argument("--seconds", type=float, default=1800.0,
help="live capture length (default 1800)")
parser.add_argument("--sample-rate", type=float, default=30.0,
help="live polling rate Hz (default 30)")
parser.add_argument("--config", default="box", help="sensor config for --live")
parser.add_argument("--backend", default="cpu", help="poisson backend for --live")
parser.add_argument("--unit", choices=("raw", "N", "both"), default="both")
parser.add_argument("--start", type=float, default=None, help="window start (s)")
parser.add_argument("--stop", type=float, default=None, help="window stop (s)")
parser.add_argument("--warmup", type=float, default=0.0,
help="discard this many seconds from the window start")
parser.add_argument("--out", default=None,
help="write the per-axis table to this CSV path, or the "
"capture directory when used with --live")
parser.add_argument("--verbose", action="store_true")
args = parser.parse_args(argv)
if args.live:
return _capture_live(args)
return _analyse(_load_rows(args.recording), args)
if __name__ == "__main__":
sys.exit(main())
+9 -5
View File
@@ -1,14 +1,18 @@
import os import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2
import orisys
import numpy as np
import time
import argparse import argparse
import time
import cv2
import numpy as np
import pyqtgraph as pg import pyqtgraph as pg
from PyQt5.QtGui import QImage, QFont from PyQt5.QtGui import QFont, QImage
from pyqtgraph.Qt import QtGui from pyqtgraph.Qt import QtGui
import orisys
def make_gradient_brush(color_hex, max_alpha=150): def make_gradient_brush(color_hex, max_alpha=150):
""" """
创建一个从上到下渐变的笔刷 (兼容 PyQt5 和 PyQt6) 创建一个从上到下渐变的笔刷 (兼容 PyQt5 和 PyQt6)
+16 -2
View File
@@ -1,5 +1,5 @@
{ {
"schema_version": 1, "schema_version": 2,
"unit": "N", "unit": "N",
"channels": { "channels": {
"normal": { "normal": {
@@ -14,8 +14,22 @@
[62000.0, 2.377] [62000.0, 2.377]
] ]
}, },
"shear": { "shear_x": {
"method": "piecewise_linear", "method": "piecewise_linear",
"raw_key": "fshearx",
"input": "components",
"signed": true,
"extrapolate": true,
"clamp_output_min": 0.0,
"points": [
[1000.0, 0.0],
[36000.0, 1.19],
[68000.0, 2.06]
]
},
"shear_y": {
"method": "piecewise_linear",
"raw_key": "fsheary",
"input": "components", "input": "components",
"signed": true, "signed": true,
"extrapolate": true, "extrapolate": true,
+10 -3
View File
@@ -29,14 +29,21 @@
"loop_closure_interval": 1, "loop_closure_interval": 1,
"algorithm": "dis", "algorithm": "dis",
"dis_preset": "medium", "dis_preset": "medium",
"dis_mode": 3, "dis_mode": 1,
"algorithm_list": ["raft", "dis"], "algorithm_list": ["raft", "dis"],
"hhd_downsample": 3 "hhd_downsample": 3
}, },
"filters": {
"flow_ema_alpha": 1.0,
"force_median_window": 1,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
},
"contact": { "contact": {
"min_contour_area": 3000, "min_contour_area": 3000,
"flow_threshold":5, "flow_threshold":0.2,
"div_threshold":0.05, "div_threshold":0.007,
"expansion_threshold": 0.01, "expansion_threshold": 0.01,
"kernel_size": [5,5], "kernel_size": [5,5],
"denoise_sigma": 12 "denoise_sigma": 12
+9 -2
View File
@@ -22,12 +22,19 @@
"dis_preset": "medium", "dis_preset": "medium",
"dis_mode": 1, "dis_mode": 1,
"algorithm_list": ["raft", "dis"], "algorithm_list": ["raft", "dis"],
"hhd_downsample": 1 "hhd_downsample": 2
},
"filters": {
"flow_ema_alpha": 1.0,
"force_median_window": 1,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
}, },
"contact": { "contact": {
"min_contour_area": 3000, "min_contour_area": 3000,
"flow_threshold": 0.2, "flow_threshold": 0.2,
"div_threshold": 0.022, "div_threshold": 0.007,
"expansion_threshold": 0.01, "expansion_threshold": 0.01,
"kernel_size": [5, 5], "kernel_size": [5, 5],
"denoise_sigma": 12 "denoise_sigma": 12
+10 -3
View File
@@ -29,14 +29,21 @@
"loop_closure_interval": 1, "loop_closure_interval": 1,
"algorithm": "dis", "algorithm": "dis",
"dis_preset": "medium", "dis_preset": "medium",
"dis_mode": 3, "dis_mode": 1,
"algorithm_list": ["raft", "dis"], "algorithm_list": ["raft", "dis"],
"hhd_downsample": 3 "hhd_downsample": 3
}, },
"filters": {
"flow_ema_alpha": 1.0,
"force_median_window": 1,
"force_lowpass_cutoff_hz": 0.0,
"force_lowpass_order": 2,
"force_lowpass_sample_rate_hz": "auto"
},
"contact": { "contact": {
"min_contour_area": 3000, "min_contour_area": 3000,
"flow_threshold":5, "flow_threshold":0.2,
"div_threshold":0.05, "div_threshold":0.007,
"expansion_threshold": 0.01, "expansion_threshold": 0.01,
"kernel_size": [5,5], "kernel_size": [5,5],
"denoise_sigma": 12 "denoise_sigma": 12
+1 -3
View File
@@ -5,9 +5,7 @@ Historical import path ``from orisys.Stitcher import Stitcher`` remains supporte
from .stitch import Stitcher from .stitch import Stitcher
from .stitch.backends.numpy_backend import blend_kernel_numpy as _blend_kernel_numpy from .stitch.backends.numpy_backend import blend_kernel_numpy as _blend_kernel_numpy
from .stitch.config import BLEND_MODES as _BLEND_MODES from .stitch.config import numba_blend_available
from .stitch.config import DEFAULT_BLEND_MODE as _DEFAULT_BLEND_MODE
from .stitch.config import numba_blend_available, resolve_blend_mode as _resolve_blend_mode
# Backward-compatible aliases used by older call sites / tests. # Backward-compatible aliases used by older call sites / tests.
_blend_kernel = _blend_kernel_numpy _blend_kernel = _blend_kernel_numpy
+29 -7
View File
@@ -5,16 +5,20 @@ A Python library for processing and analyzing optical tactile sensor data,
including image stitching, optical flow tracking, and deformation decomposition. including image stitching, optical flow tracking, and deformation decomposition.
""" """
__version__ = "0.5.0" __version__ = "0.5.4"
__author__ = "Orisys" __author__ = "Orisys"
from .sensor import Sensor from . import calibration, capture, configs, finger, signal_processing, util
from . import util from .export import StreamRecorder, mosaic_videos
from . import configs
from . import finger
from . import signal_processing
from . import capture
from .frame_processors import FFCFrameProcessor from .frame_processors import FFCFrameProcessor
from .pipeline import (
FrameSnapshot,
PauseLevel,
PipelineStats,
ProducedFrame,
StreamPipeline,
)
from .sensor import Sensor
CompoundEyeSensor = Sensor CompoundEyeSensor = Sensor
CameraSensor = Sensor CameraSensor = Sensor
@@ -22,6 +26,8 @@ CameraSensor = Sensor
# Re-export list_configs as list_config for convenience # Re-export list_configs as list_config for convenience
list_config = configs.list_config list_config = configs.list_config
export_config = configs.export_config export_config = configs.export_config
export_sensor_bundle = configs.export_sensor_bundle
export_sensor_bundles = configs.export_sensor_bundles
__all__ = [ __all__ = [
# class # class
@@ -41,9 +47,25 @@ __all__ = [
# capture helpers # capture helpers
'capture', 'capture',
# pipelined multi-stream processing
'StreamPipeline',
'FrameSnapshot',
'ProducedFrame',
'PipelineStats',
'PauseLevel',
# offline recording of arrow overlays
'StreamRecorder',
'mosaic_videos',
# force calibration
'calibration',
# configs utilities functions # configs utilities functions
'list_config', 'list_config',
'export_config', 'export_config',
'export_sensor_bundle',
'export_sensor_bundles',
# test # test
'FFCFrameProcessor' 'FFCFrameProcessor'
+1 -1
View File
@@ -63,8 +63,8 @@ except ImportError:
try: try:
import pyopencl as cl import pyopencl as cl
import pyopencl.array as cla import pyopencl.array as cla
from pyvkfft.fft import rfftn as cl_rfftn
from pyvkfft.fft import irfftn as cl_irfftn from pyvkfft.fft import irfftn as cl_irfftn
from pyvkfft.fft import rfftn as cl_rfftn
_opencl_cl = cl _opencl_cl = cl
_opencl_cla = cla _opencl_cla = cla
_opencl_rfftn = cl_rfftn _opencl_rfftn = cl_rfftn
File diff suppressed because it is too large Load Diff
+15 -13
View File
@@ -16,6 +16,7 @@ from .discovery import (
pick_default_camera, pick_default_camera,
) )
from .source import ( from .source import (
CapturedFrame,
CaptureSource, CaptureSource,
CaptureWorker, CaptureWorker,
is_live_camera, is_live_camera,
@@ -25,21 +26,22 @@ from .source import (
) )
__all__ = [ __all__ = [
"MAX_SIMULTANEOUS_CAPTURES",
"DEFAULT_MAX_CAMERA_INDEX",
"DEFAULT_LAPTOP_WEBCAM_PATTERNS", "DEFAULT_LAPTOP_WEBCAM_PATTERNS",
"CaptureConfig", "DEFAULT_MAX_CAMERA_INDEX",
"CaptureHealth", "MAX_SIMULTANEOUS_CAPTURES",
"CaptureStatus",
"ReconnectPolicy",
"CaptureSource",
"CaptureWorker",
"CameraInfo",
"CameraFilter", "CameraFilter",
"list_cameras", "CameraInfo",
"pick_default_camera", "CaptureConfig",
"normalize_source", "CapturedFrame",
"CaptureHealth",
"CaptureSource",
"CaptureStatus",
"CaptureWorker",
"ReconnectPolicy",
"is_live_camera", "is_live_camera",
"open_opencv_capture", "list_cameras",
"normalize_source",
"open_capture", "open_capture",
"open_opencv_capture",
"pick_default_camera",
] ]
+18 -17
View File
@@ -2,9 +2,10 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass, field from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import Any, Optional, Sequence, Tuple, Union from typing import Any, Union
MAX_SIMULTANEOUS_CAPTURES = 10 MAX_SIMULTANEOUS_CAPTURES = 10
@@ -44,11 +45,11 @@ class ReconnectPolicy:
""" """
enabled: bool = False enabled: bool = False
max_attempts: Optional[int] = None max_attempts: int | None = None
initial_backoff_s: float = 0.25 initial_backoff_s: float = 0.25
max_backoff_s: float = 5.0 max_backoff_s: float = 5.0
fail_threshold: int = 5 fail_threshold: int = 5
stale_timeout_s: Optional[float] = 2.0 stale_timeout_s: float | None = 2.0
def next_backoff(self, attempt: int) -> float: def next_backoff(self, attempt: int) -> float:
"""Exponential backoff capped at ``max_backoff_s`` (attempt is 0-based).""" """Exponential backoff capped at ``max_backoff_s`` (attempt is 0-based)."""
@@ -62,19 +63,19 @@ class CaptureConfig:
source: SourceType source: SourceType
backend: str = "auto" backend: str = "auto"
width: Optional[int] = None width: int | None = None
height: Optional[int] = None height: int | None = None
fps: Optional[float] = None fps: float | None = None
fourcc: Optional[str] = "MJPG" fourcc: str | None = "MJPG"
background: bool = True background: bool = True
reconnect: ReconnectPolicy = field(default_factory=ReconnectPolicy) reconnect: ReconnectPolicy = field(default_factory=ReconnectPolicy)
verbose: bool = False verbose: bool = False
# Optional discovery filters (used by list/open helpers, not open itself). # Optional discovery filters (used by list/open helpers, not open itself).
name_include: Optional[Sequence[str]] = None name_include: Sequence[str] | None = None
name_exclude: Optional[Sequence[str]] = None name_exclude: Sequence[str] | None = None
vid_pid_allowlist: Optional[Sequence[Tuple[int, int]]] = None vid_pid_allowlist: Sequence[tuple[int, int]] | None = None
min_width: Optional[int] = None min_width: int | None = None
min_height: Optional[int] = None min_height: int | None = None
def __post_init__(self) -> None: def __post_init__(self) -> None:
if not isinstance(self.reconnect, ReconnectPolicy): if not isinstance(self.reconnect, ReconnectPolicy):
@@ -86,10 +87,10 @@ class CaptureConfig:
source: SourceType, source: SourceType,
camera: dict[str, Any], camera: dict[str, Any],
*, *,
reconnect: Optional[ReconnectPolicy] = None, reconnect: ReconnectPolicy | None = None,
verbose: bool = False, verbose: bool = False,
**kwargs: Any, **kwargs: Any,
) -> "CaptureConfig": ) -> CaptureConfig:
"""Build config from a sensor JSON ``camera`` section.""" """Build config from a sensor JSON ``camera`` section."""
# Accept deprecated queue/worker kwargs from older callers without error. # Accept deprecated queue/worker kwargs from older callers without error.
kwargs.pop("mode", None) kwargs.pop("mode", None)
@@ -117,9 +118,9 @@ class CaptureStatus:
"""Snapshot of capture health for diagnostics / multi-camera UIs.""" """Snapshot of capture health for diagnostics / multi-camera UIs."""
health: CaptureHealth = CaptureHealth.CLOSED health: CaptureHealth = CaptureHealth.CLOSED
source: Optional[SourceType] = None source: SourceType | None = None
last_error: Optional[str] = None last_error: str | None = None
last_frame_time: Optional[float] = None last_frame_time: float | None = None
last_seq: int = 0 last_seq: int = 0
dropped_frames: int = 0 dropped_frames: int = 0
reconnect_attempts: int = 0 reconnect_attempts: int = 0

Some files were not shown because too many files have changed in this diff Show More