Browse a long recording from disk instead of loading it into memory
What
- record/index.py: RecordingIndex streams forces.csv once and keeps only the
small things — one time, one validity flag, one frame id per row plus a byte
offset every 4096 rows (~6.5 MB per 500k rows). row_range() bisects the times,
read_rows() seeks to the nearest checkpoint and scans forward, and overview()
folds every row into its bucket during the same pass to build a per-axis
min/max envelope.
- read_rows(max_points, unit) decimates **while streaming**: each bucket keeps
the extreme row of every axis in the requested unit, so a long interval never
has to exist in memory in full and a shear spike survives a flat normal channel.
- RecordingReader: info() decimates above max_rows (default 200k) and marks
itself decimated, with total_rows and a duration taken from the frame index
rather than from the view; window()/window_all()/overview()/statistics() read
the interval being asked about. list/window semantics unchanged for small
recordings.
- force/source.py: StoreCurveSource (live memory) and RecordingCurveSource (disk
interval + cached envelope) implement one interface — extent, window,
envelope, statistics, duration_s — so ForcePlotsPanel asks a *source* instead
of knowing where the data lives.
- force_plots: reads through the source (window, envelope, statistics, extent),
which also replaces its last direct store accesses.
- playback_dialog: draws the whole-session view through window_all() in the unit
being shown (bounded by CHART_MAX_POINTS), and says so when the view is
decimated ("显示 N / 共 M 行") instead of letting the operator count points.
- experiment._run: a failure that lands while the files are opening can no longer
be overwritten by the writer's own transition to "recording" (found by a flaky
test; the state check now covers STATE_FAILED).
Why
The store releases old blocks above ~4.6 h and the replay loaded every CSV row
into Python lists before drawing anything, so neither the live browser nor the
replay could show an earlier interval of a long recording. The index makes the
disk the data source and memory a cache of what is on screen.
Verification
- shared/tests/test_history_source.py (new, 8 cases) over a 12 000-row recording:
the index reports its rows and reads a bounded interval; a large recording is
marked decimated with its true duration and total rows; a small one is not; an
interval read sees the exact rows including a single peak; a decimated window
keeps the peak; the envelope fits the whole session in ≤200 points, keeps the
spike and is cached; the disk and memory sources agree on extent, values and
statistics; the whole session can be read without holding it all. Shared
suites: 146 tests, 0 failures (three consecutive runs).
- apps/studio/tools/ui_check.py: the replay section now checks the index row
count, that an interval read spans only the interval, the envelope size and
total_rows. 324 checks, 0 failures.
- apps/studio/tests/test_ui_smoke.py: 43 tests, 36 pass; the 7 failures are the
unchanged sandbox/harness artifacts.
- D:/orisys_ws/_review_phase12/reproduce.py: all ten lines unchanged;
compileall, locale and prefs parity clean.
- apps/studio/docs/record-channel.md §8 documents the index, the reader API and
the unified source, and lists what is still hardware work.
Not in this change
- Maximum RSS / backlog / loss measurements at real resolution on the target ARM
board (needs the hardware), and range CSV export.
This commit is contained in:
Binary file not shown.
|
Before Width: | Height: | Size: 2.0 MiB After Width: | Height: | Size: 2.0 MiB |
@@ -115,6 +115,8 @@
|
||||
"box_zoom_tip": "Switch to rubber-band zoom (drag a region on the chart)",
|
||||
"selection": "Selection {duration}s / {samples} samples — {detail}",
|
||||
"selection_empty": "No valid samples in the selection",
|
||||
"value_unavailable": "n/a",
|
||||
"value_unavailable_tip": "No curve is loaded for {channel}, so it has no value in this unit (raw counts are never shown in its place)",
|
||||
"browse_only_tip": "Drag or scroll on the chart to browse history"
|
||||
},
|
||||
"layout": {
|
||||
@@ -154,14 +156,21 @@
|
||||
"depth_state_approximate": "No reset or pre-roll history for this segment: the algorithm state is an approximation",
|
||||
"depth_warnings": "Recomputation notes: {detail}",
|
||||
"warnings": "Replay notes: {detail}",
|
||||
"decimated": "This recording is long, so the curves are a decimated view ({shown} of {total} rows; intervals are read from disk on demand)",
|
||||
"warn_config_changed": "the config differs from recording time, so the recomputed depth may differ",
|
||||
"warn_sdk_changed": "the SDK version differs from recording time, so the recomputation may differ",
|
||||
"warn_config_missing": "no config hash was recorded, so recomputation cannot be verified",
|
||||
"warn_depth_recomputed": "depth comes from offline recomputation, not from stored data",
|
||||
"warn_rows_skipped": "some unparsable CSV rows were skipped",
|
||||
"warn_frames_dropped": "frames were lost to a full queue; only their index rows remain (shown as \"not recorded\")",
|
||||
"warn_out_of_order": "this recording was written by an older version: its rows are not in time order, so the replay re-sorts them and window edges may differ",
|
||||
"warn_state_across_segment": "the algorithm state cannot be carried across a segment",
|
||||
"warn_state_approximate": "the algorithm state is an approximation"
|
||||
"warn_state_approximate": "the algorithm state is an approximation",
|
||||
"warn_state_approximate_after_rewind": "the video was reopened to reach an earlier frame, so the state may differ",
|
||||
"warn_state_too_far_back": "the zeroing is too far back: reading there would exceed the replay budget, so the state is approximate",
|
||||
"warn_reset_frame_not_indexed": "the zeroing frame is not in the frame index, so the state cannot be rebuilt from it",
|
||||
"warn_reset_frame_not_in_video": "the zeroing frame never reached the video, so the state cannot be rebuilt from it",
|
||||
"warn_reset_not_applied": "the replay could not re-apply the zeroing, so the state may differ"
|
||||
},
|
||||
"record": {
|
||||
"label": "Record",
|
||||
@@ -177,6 +186,8 @@
|
||||
"start_failed": "Could not start recording: {error}",
|
||||
"finished": "Recording finished — {frames} frames / {segments} segment(s) in {path}",
|
||||
"finished_with_problems": "Recording finished with problems: {detail}",
|
||||
"closing": "Closing the recording files…",
|
||||
"failed_state": "The recording failed (see the summary and the log)",
|
||||
"alarm": "Recording problem: {detail}"
|
||||
},
|
||||
"cal": {
|
||||
@@ -204,8 +215,12 @@
|
||||
"open_record_tip": "Reopen a saved calibration session to inspect every point's samples and diagnostics",
|
||||
"open_record_filter": "Calibration records (*.json)",
|
||||
"open_record_ok": "Loaded {path} ({points} point(s))",
|
||||
"open_record_device": "Recorded on: {device}",
|
||||
"open_record_warnings": "Note: {detail}",
|
||||
"open_record_failed": "Could not open the calibration record: {error}",
|
||||
"apply_mismatch_title": "Calibration does not match the target device",
|
||||
"apply_mismatch": "This coefficient file belongs to: {device}\nMismatch: {detail}\n\nCurrent target: {sensor_type} / {config}\n\nApplying it would load coefficients that were not measured on this device.",
|
||||
"apply_mismatch_skipped": "Apply cancelled (the calibration belongs to another device)",
|
||||
"sn": "SN",
|
||||
"sn_placeholder": "Serial number (optional)",
|
||||
"live_raw_label": "Live raw",
|
||||
@@ -255,11 +270,13 @@
|
||||
"export_local": "Also wrote {path}",
|
||||
"export_warnings": "Skipped: {detail}",
|
||||
"export_failed": "Export failed: {error}",
|
||||
"export_no_model": "Nothing to export: {detail}",
|
||||
"apply_title": "Apply calibration",
|
||||
"apply_confirm": "Apply the calibration just exported to the current device?\n\nSensor type: {sensor_type}\nConfig: {config}\n\nChoosing No keeps the exported file and leaves the running device untouched.",
|
||||
"apply_skipped": "Exported; not applied to the device",
|
||||
"apply_pending": "Requested calibration for {sensor_type}; waiting for the device to confirm…",
|
||||
"applied": "Device confirmed the calibration is in effect (version {version})",
|
||||
"applied_none": "Device confirmed: no calibration is loaded (the column stays empty)",
|
||||
"export_only": "Exported coefficients to {path} (not applied to the device)",
|
||||
"apply_no_target": "No device to apply to (run an acquisition first)"
|
||||
},
|
||||
@@ -286,6 +303,7 @@
|
||||
"video_ended": "Video ended",
|
||||
"camera_disconnected": "Camera disconnected",
|
||||
"reset_ok": "Tracker reset",
|
||||
"reset_pending": "Zeroing requested; waiting for the acquisition thread to confirm…",
|
||||
"finger_run_unavailable": "Run is not available for this sensor type",
|
||||
"video_required": "Select a video file first",
|
||||
"config_missing": "Sensor config not found for {type}",
|
||||
|
||||
@@ -115,6 +115,8 @@
|
||||
"box_zoom_tip": "切换为框选放大(拖动图表框选区域)",
|
||||
"selection": "选区 {duration}s / {samples} 点 — {detail}",
|
||||
"selection_empty": "选区无有效样本",
|
||||
"value_unavailable": "不可用",
|
||||
"value_unavailable_tip": "{channel} 轴未加载标定曲线,因此没有该单位的数值(不会用原始值冒充)",
|
||||
"browse_only_tip": "拖动或滚轮即可进入历史浏览"
|
||||
},
|
||||
"layout": {
|
||||
@@ -154,14 +156,21 @@
|
||||
"depth_state_approximate": "本段缺少归零或预卷历史,算法参考状态为近似重建",
|
||||
"depth_warnings": "重算提示: {detail}",
|
||||
"warnings": "回放提示: {detail}",
|
||||
"decimated": "该录制较长,曲线为降采样视图(显示 {shown} / 共 {total} 行;区间数据按需从磁盘读取)",
|
||||
"warn_config_changed": "配置文件与录制时不一致,重算出的深度可能与当时不同",
|
||||
"warn_sdk_changed": "SDK 版本与录制时不一致,重算结果可能不同",
|
||||
"warn_config_missing": "找不到配置哈希,无法校验重算一致性",
|
||||
"warn_depth_recomputed": "深度由离线重算得到,不是当时保存的数据",
|
||||
"warn_rows_skipped": "有无法解析的 CSV 行被跳过",
|
||||
"warn_frames_dropped": "有帧在队列溢出时丢失,仅保留索引记录(回放显示为“该帧未录制”)",
|
||||
"warn_out_of_order": "该录制由旧版本写入,行序不是时间序;回放已按时间重排,区间边界可能与当时不同",
|
||||
"warn_state_across_segment": "重算不能跨分段延续算法状态",
|
||||
"warn_state_approximate": "算法参考状态为近似重建"
|
||||
"warn_state_approximate": "算法参考状态为近似重建",
|
||||
"warn_state_approximate_after_rewind": "为回到更早的帧重新打开了视频,参考状态可能偏离当时",
|
||||
"warn_state_too_far_back": "归零点过于久远,需要读取的帧数超出上限,已按近似状态重建",
|
||||
"warn_reset_frame_not_indexed": "归零帧不在帧索引中,无法从该点重建状态",
|
||||
"warn_reset_frame_not_in_video": "归零帧未写入视频,无法从该点重建状态",
|
||||
"warn_reset_not_applied": "重放无法重新执行归零,参考状态可能偏离当时"
|
||||
},
|
||||
"record": {
|
||||
"label": "记录",
|
||||
@@ -177,6 +186,8 @@
|
||||
"start_failed": "无法开始记录: {error}",
|
||||
"finished": "记录完成 — {frames} 帧 / {segments} 段,已写入 {path}",
|
||||
"finished_with_problems": "记录完成,但存在问题: {detail}",
|
||||
"closing": "正在关闭记录文件…",
|
||||
"failed_state": "记录已失败(详见摘要与日志)",
|
||||
"alarm": "记录异常: {detail}"
|
||||
},
|
||||
"cal": {
|
||||
@@ -204,8 +215,12 @@
|
||||
"open_record_tip": "重新打开已保存的标定会话,检查每个点的完整采样与诊断状态",
|
||||
"open_record_filter": "标定记录 (*.json)",
|
||||
"open_record_ok": "已载入 {path}({points} 个点)",
|
||||
"open_record_device": "记录设备: {device}",
|
||||
"open_record_warnings": "注意: {detail}",
|
||||
"open_record_failed": "打开标定记录失败: {error}",
|
||||
"apply_mismatch_title": "标定与目标设备不匹配",
|
||||
"apply_mismatch": "该标定文件属于: {device}\n不匹配项: {detail}\n\n当前目标: {sensor_type} / {config}\n\n继续应用会把不适用于本设备的系数写入运行中的设备。",
|
||||
"apply_mismatch_skipped": "已取消应用(标定身份与目标设备不匹配)",
|
||||
"sn": "序列号",
|
||||
"sn_placeholder": "序列号(可选)",
|
||||
"live_raw_label": "实时原始值",
|
||||
@@ -255,11 +270,13 @@
|
||||
"export_local": "同时写入 {path}",
|
||||
"export_warnings": "跳过: {detail}",
|
||||
"export_failed": "导出失败: {error}",
|
||||
"export_no_model": "没有可导出的模型: {detail}",
|
||||
"apply_title": "应用标定",
|
||||
"apply_confirm": "将刚导出的标定应用到当前设备?\n\n设备类型: {sensor_type}\n配置: {config}\n\n选择「否」只保留导出文件,不改动运行中的设备。",
|
||||
"apply_skipped": "已导出,未应用到设备",
|
||||
"apply_pending": "已请求把标定应用到 {sensor_type},等待设备确认…",
|
||||
"applied": "设备已确认标定生效(版本 {version})",
|
||||
"applied_none": "设备已确认:当前没有加载标定(该项留空)",
|
||||
"export_only": "已导出系数 {path}(未应用到设备)",
|
||||
"apply_no_target": "没有可应用的设备(请先运行一次采集)"
|
||||
},
|
||||
@@ -286,6 +303,7 @@
|
||||
"video_ended": "视频已结束",
|
||||
"camera_disconnected": "摄像机已断开",
|
||||
"reset_ok": "追踪器已重置",
|
||||
"reset_pending": "已请求归零,等待采集线程确认…",
|
||||
"finger_run_unavailable": "此传感器类型暂不支持运行",
|
||||
"video_required": "请先选择视频文件",
|
||||
"config_missing": "未找到传感器配置: {type}",
|
||||
|
||||
@@ -293,9 +293,44 @@ outputs/recordings/<UTC 时间戳>_session<NNNN>/
|
||||
|
||||
---
|
||||
|
||||
## 8. 尚未完成(第三阶段及以后)
|
||||
## 8. 长历史浏览:按需读盘而不是整段载入
|
||||
|
||||
`record/index.py` 的 `RecordingIndex` 是回放的数据源索引:
|
||||
|
||||
- 流式扫一遍 `forces.csv`,只把**小东西**留在内存:每行一个时间(float64)、一个
|
||||
valid 标志、一个帧号,以及每 `CHECKPOINT_ROWS`(4096)行的字节偏移。500k 行约
|
||||
6.5 MB(`SessionStore` 的默认上限正好对应 30 Hz 下约 4.6 小时)。
|
||||
- `row_range(t0, t1)` 用二分定位行区间,`read_rows(start, stop)` 先 `seek` 到最近的
|
||||
检查点再向前扫描——读一个 10 秒区间就是 10 秒的解析量,不是整段。
|
||||
区间会额外带上 `t0` 之前的那一行,绘制时才能画出跨过左边界的线段。
|
||||
- `overview(buckets)` 在**同一次流式扫描**里把每行折进所属桶,得到每桶每轴的
|
||||
min/max/count。全程色带因此不会漏掉某桶里的尖峰(按行抽样会漏)。
|
||||
|
||||
`ReadingReader` 相应变化:
|
||||
|
||||
| 方法 | 作用 |
|
||||
|---|---|
|
||||
| `info()` | 行数超过 `max_rows`(默认 200k)时给出**降采样视图**,并置 `decimated=True`;`duration_s` 始终取自帧索引,不取自视图 |
|
||||
| `window(t0, t1, max_points=…)` | 区间精确读取(可再降采样),尖峰按轴保留 |
|
||||
| `overview(unit, buckets)` | 全程包络,按 (unit, buckets) 缓存 |
|
||||
| `statistics(t0, t1, unit)` | 只算该区间的统计量 |
|
||||
|
||||
`force/source.py` 把两种来源统一成同一接口:`StoreCurveSource`(实时内存块)与
|
||||
`RecordingCurveSource`(磁盘区间 + 缓存包络),都提供
|
||||
`extent() / window() / envelope() / statistics() / duration_s()`;`ForcePlotsPanel`
|
||||
只认这个接口,因此同一套浏览逻辑既能看内存会话也能看磁盘录制。
|
||||
|
||||
回放窗口在曲线被降采样时会明确写出「显示 N / 共 M 行」,而不是让操作员自己数点。
|
||||
|
||||
> 未做的验收项:真实分辨率与目标 ARM 板上的最大 RSS、积压与丢失测试(需要硬件),
|
||||
> 以及选中时间段的 CSV 导出。
|
||||
|
||||
---
|
||||
|
||||
## 9. 尚未完成(第三阶段及以后)
|
||||
|
||||
- PyOpenGL 三维显示(`mesh3d_panel.py` 目前仍是占位符,不是已有实现)。
|
||||
- CSV 的「导出选中时间段」(整段导出已随录制落盘;区间导出待做)。
|
||||
- 标定的稳定后采样(当前按固定时长采样,稳定性以 σ 呈现)。
|
||||
- 在目标 ARM 板上用真实分辨率测长时录制的最大 RSS、队列积压与丢失。
|
||||
|
||||
|
||||
@@ -34,10 +34,11 @@ from orisys_app_shared.acquisition.constants import (
|
||||
)
|
||||
from orisys_app_shared.force import select_display_forces
|
||||
from orisys_app_shared.force.calibration_recorder import CalibrationRecorder
|
||||
from orisys_app_shared.force.session import SessionSample, SessionStore
|
||||
from orisys_app_shared.force.session import SessionStore
|
||||
from orisys_app_shared.paths import recordings_dir
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig
|
||||
from orisys_app_shared.record import RecordConfig
|
||||
from orisys_app_shared.record.sample import to_record_sample
|
||||
from orisys_studio.recording import RecordingCoordinator, record_config_from_prefs
|
||||
from orisys_app_shared.i18n import set_locale, t
|
||||
from orisys_app_shared.logging import report_error
|
||||
from orisys_app_shared.paths import logo_path, sensor_json_path
|
||||
@@ -133,20 +134,20 @@ class StudioMainWindow(QMainWindow):
|
||||
on_finished=self._on_calibration_point,
|
||||
on_cancelled=self._on_calibration_cancelled,
|
||||
)
|
||||
self.recorder = ExperimentRecorder()
|
||||
self.recorder.set_alarm_handler(self._on_record_alarm)
|
||||
self._record_alarms: list[str] = []
|
||||
self._record_last_report = None
|
||||
self._record_fps = 30.0
|
||||
# The record channel's mechanics (recorder lifecycle, metadata, the
|
||||
# per-packet fan-out, confirmed events) live in the coordinator; this
|
||||
# window keeps the buttons, the menus and the status messages.
|
||||
self.recording = RecordingCoordinator(
|
||||
session_store=self.session_store,
|
||||
cal_recorder=self.cal_recorder,
|
||||
)
|
||||
self.recorder = self.recording.recorder
|
||||
self._session_counter = 0
|
||||
self._playback = None
|
||||
self._record_error_count = 0
|
||||
self._dropped_reported = 0
|
||||
self._active_calibration_id = ""
|
||||
#: Last coefficients file written by the calibration panel (not applied).
|
||||
self._last_calibration_export = ""
|
||||
#: Confirmed calibration changes, handed over from the acquisition thread.
|
||||
self._pending_calibration_events: list[tuple[str, str, int]] = []
|
||||
self._depth_range_value: tuple[float, float] | None = None
|
||||
self._last_depth_frame = None
|
||||
|
||||
@@ -249,14 +250,28 @@ class StudioMainWindow(QMainWindow):
|
||||
worker.add_sink(self._on_packet_recorded)
|
||||
|
||||
# ------------------------------------------------------- experiment record
|
||||
def _on_record_alarm(self, message: str) -> None:
|
||||
"""Called from the writer thread. Queue it; never touch Qt here."""
|
||||
logger.error("Recording alarm: %s", message)
|
||||
self._record_alarms.append(str(message))
|
||||
@property
|
||||
def _record_fps(self) -> float:
|
||||
return self.recording.record_fps
|
||||
|
||||
@property
|
||||
def _active_calibration_id(self) -> str:
|
||||
return self.recording.active_calibration
|
||||
|
||||
@_active_calibration_id.setter
|
||||
def _active_calibration_id(self, value: str) -> None:
|
||||
self.recording.set_active_calibration(value)
|
||||
|
||||
@property
|
||||
def _record_last_report(self):
|
||||
return self.recording.last_report
|
||||
|
||||
@property
|
||||
def _record_report_pending(self) -> bool:
|
||||
return self.recording.report_pending
|
||||
|
||||
def _drain_record_alarms(self) -> None:
|
||||
while self._record_alarms:
|
||||
message = self._record_alarms.pop(0)
|
||||
for message in self.recording.drain_alarms():
|
||||
self._set_phase("error")
|
||||
self.statusBar().showMessage(t("record.alarm", detail=message))
|
||||
|
||||
@@ -274,16 +289,13 @@ class StudioMainWindow(QMainWindow):
|
||||
self.statusBar().showMessage(t("record.requires_run"))
|
||||
return
|
||||
mode = self.toolbar.record_mode()
|
||||
config = RecordConfig(
|
||||
video=(mode == "video_data"),
|
||||
lossless_frames=bool(self.prefs.get("record_lossless_frames", False)),
|
||||
fps=float(self._record_fps),
|
||||
segment_seconds=float(self.prefs.get("record_segment_seconds", 300.0)),
|
||||
config = record_config_from_prefs(
|
||||
self.prefs, mode=mode, fps=float(self._record_fps)
|
||||
)
|
||||
try:
|
||||
directory = self.recorder.start(
|
||||
directory = self.recording.start(
|
||||
root=recordings_dir(),
|
||||
session_id=int(self.session_store.session_id),
|
||||
config=config,
|
||||
metadata=self._build_session_metadata(config),
|
||||
calibration_id=self._active_calibration_id,
|
||||
calibration_version=self._active_calibration_id,
|
||||
@@ -297,21 +309,41 @@ class StudioMainWindow(QMainWindow):
|
||||
self.statusBar().showMessage(t("record.start_failed", error=str(exc)))
|
||||
logger.exception("Could not start recording")
|
||||
return
|
||||
self._record_last_report = None
|
||||
self.recorder.log_event("record_started", f"mode={mode}")
|
||||
self.toolbar.set_record_status(Path(directory).name)
|
||||
self.statusBar().showMessage(t("record.started", mode=t(f"record.mode_{mode}")))
|
||||
logger.info("Recording started: %s (mode=%s)", directory, mode)
|
||||
|
||||
def _stop_recording(self, *, wait: bool = True) -> None:
|
||||
if not self.recorder.is_recording:
|
||||
self.toolbar.set_record_status("")
|
||||
return
|
||||
self.recorder.log_event("record_stopped", "")
|
||||
report = self.recorder.stop(wait=wait)
|
||||
self._record_last_report = report
|
||||
"""Stop the recording and publish what actually happened.
|
||||
|
||||
A recording that already failed on its own is **not** a no-op: its report
|
||||
still has to be consumed, or the button keeps claiming a recording is
|
||||
running and the failure is never shown.
|
||||
|
||||
``wait=False`` returns as soon as the stop is requested; the writer owns
|
||||
the completion and the UI timer publishes it (see ``_poll_record_report``).
|
||||
"""
|
||||
if self.recorder.directory is None and self.recorder.report is None:
|
||||
# Nothing was ever started: keep the widget state consistent anyway.
|
||||
self.toolbar.set_record_status("")
|
||||
self.toolbar.set_record_checked(False)
|
||||
return
|
||||
self._publish_record_report(self.recording.stop(wait=wait))
|
||||
|
||||
def _publish_record_report(self, report) -> None:
|
||||
"""Update the button and the summary from a recording report."""
|
||||
self.toolbar.set_record_status("")
|
||||
self.toolbar.set_record_checked(False)
|
||||
if report is None:
|
||||
return
|
||||
if not report.completed:
|
||||
# The writer is still closing its files. Never present that as a
|
||||
# finished experiment; the timer publishes the real report.
|
||||
self.statusBar().showMessage(t("record.closing"))
|
||||
return
|
||||
if report.stats.failed:
|
||||
self._set_phase("error")
|
||||
if report.problems:
|
||||
self.statusBar().showMessage(
|
||||
t("record.finished_with_problems", detail="; ".join(report.problems))
|
||||
@@ -326,6 +358,12 @@ class StudioMainWindow(QMainWindow):
|
||||
)
|
||||
)
|
||||
|
||||
def _poll_record_report(self) -> None:
|
||||
"""Publish the terminal report once the writer has closed the files."""
|
||||
report = self.recording.poll()
|
||||
if report is not None:
|
||||
self._publish_record_report(report)
|
||||
|
||||
def _take_preroll(self) -> list:
|
||||
"""Frame history from the active worker, if it keeps one.
|
||||
|
||||
@@ -333,15 +371,17 @@ class StudioMainWindow(QMainWindow):
|
||||
recording follows, so a stale history can never be attached to a later
|
||||
recording.
|
||||
"""
|
||||
worker = self._active_worker
|
||||
take = getattr(worker, "take_preroll", None)
|
||||
if not callable(take):
|
||||
return []
|
||||
try:
|
||||
return list(take())
|
||||
except Exception: # noqa: BLE001 - the history is optional, not the record
|
||||
logger.exception("Could not take the pre-roll buffer")
|
||||
return []
|
||||
return self.recording.take_preroll(self._active_worker)
|
||||
|
||||
@staticmethod
|
||||
def _device_serial(info) -> str:
|
||||
"""The sensor's own serial number, when the SDK exposes one."""
|
||||
for attribute in ("sn", "serial", "serial_number"):
|
||||
value = getattr(info, attribute, None)
|
||||
if value:
|
||||
return str(value)
|
||||
serial = getattr(info, "info", None)
|
||||
return str(getattr(serial, "SN", "") or "")
|
||||
|
||||
def _build_session_metadata(self, config: RecordConfig) -> dict:
|
||||
from orisys_app_shared import API_VERSION, __version__ as shared_version
|
||||
@@ -379,6 +419,10 @@ class StudioMainWindow(QMainWindow):
|
||||
# filled in by the acquisition thread's confirmation, so it can
|
||||
# be empty here; the CSV's per-row version is authoritative.
|
||||
"version": self._active_calibration_id,
|
||||
# The operator-entered serial of the *calibrated* device, as the
|
||||
# calibration panel has it (the device SN when the config has one).
|
||||
"sn": self.cal_panel.sn_edit.text().strip(),
|
||||
"device_sn": self._device_serial(info),
|
||||
},
|
||||
# Depth is deliberately NOT stored (decided with the team): it is a
|
||||
# function of the frame and the sensor configuration, so replay
|
||||
@@ -395,7 +439,10 @@ class StudioMainWindow(QMainWindow):
|
||||
"conventions": {
|
||||
"axis_order": ["FN", "FX", "FY"],
|
||||
"force_unit": "N when calibrated, otherwise raw counts",
|
||||
"time_base": "t_rel_s is relative to the first recorded frame",
|
||||
"time_base": (
|
||||
"t_rel_s is relative to the first recorded frame; an event "
|
||||
"that predates it has no relative base and is left empty"
|
||||
),
|
||||
"frame_id": "monotonic per acquisition session",
|
||||
},
|
||||
"filter": {
|
||||
@@ -432,25 +479,10 @@ class StudioMainWindow(QMainWindow):
|
||||
def _on_packet_recorded(self, packet) -> None:
|
||||
"""Runs on the **acquisition thread**. Must stay cheap and Qt-free.
|
||||
|
||||
The plots are allowed to drop frames; this path is not. It is also the
|
||||
reason calibration sampling no longer depends on the calibration panel
|
||||
being visible.
|
||||
|
||||
The packet is converted exactly once, by the same function the CSV writer
|
||||
uses, so the store and the file can never disagree about whether a value
|
||||
is a measurement.
|
||||
The record channel's mechanics are the coordinator's; all this keeps is
|
||||
the UI's own bookkeeping (how many readouts failed).
|
||||
"""
|
||||
record = to_record_sample(
|
||||
packet, calibration_version=self._active_calibration_id
|
||||
)
|
||||
sample = SessionSample.from_record(record)
|
||||
self.session_store.append(sample)
|
||||
self.cal_recorder.feed(sample)
|
||||
# Video + CSV + metadata share this one record path.
|
||||
self.recorder.on_packet(packet)
|
||||
fps = float(getattr(packet, "fps", 0.0) or 0.0)
|
||||
if fps > 0.0:
|
||||
self._record_fps = fps
|
||||
record = self.recording.on_packet(packet)
|
||||
if not record.valid:
|
||||
self._record_error_count += 1
|
||||
|
||||
@@ -919,11 +951,64 @@ class StudioMainWindow(QMainWindow):
|
||||
self.recorder.log_event("calibration_exported", str(path))
|
||||
self.statusBar().showMessage(t("cal.export_only", path=Path(path).name))
|
||||
|
||||
def calibration_identity(self, path: str) -> dict:
|
||||
"""What device a coefficient file says it belongs to.
|
||||
|
||||
Read with the SDK's own parser so the check sees the same document the
|
||||
device would load. A file that cannot be read yields no identity, and the
|
||||
caller then applies it without a mismatch claim.
|
||||
"""
|
||||
try:
|
||||
import orisys.calibration as sdk_calibration
|
||||
|
||||
document = sdk_calibration.load(path, strict=True)
|
||||
except Exception: # noqa: BLE001 - reported by the apply path itself
|
||||
logger.exception("Could not read calibration identity from %s", path)
|
||||
return {}
|
||||
identity = dict(getattr(document, "identity", {}) or {})
|
||||
pipeline = dict(getattr(document, "pipeline", {}) or {})
|
||||
return {
|
||||
"sensor_type": str(identity.get("sensor_type") or ""),
|
||||
"sn": str(identity.get("sn") or ""),
|
||||
"config_ref": str(pipeline.get("config_ref") or ""),
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _same_path(first: str, second: str) -> bool:
|
||||
if not first or not second:
|
||||
return True # nothing to compare: not a mismatch
|
||||
if first == second:
|
||||
return True
|
||||
try:
|
||||
return Path(first).resolve() == Path(second).resolve()
|
||||
except Exception: # noqa: BLE001 - a non-path value simply differs
|
||||
return False
|
||||
|
||||
def calibration_target_mismatch(
|
||||
self, path: str, *, sensor_type: str, config_ref: str
|
||||
) -> list[str]:
|
||||
"""Reasons this calibration does not obviously belong to this device.
|
||||
|
||||
Applying coefficients that were measured on a different sensor silently
|
||||
changes what every recorded newton value means, so the mismatch is stated
|
||||
before the confirmation, not discovered later.
|
||||
"""
|
||||
recorded = self.calibration_identity(path)
|
||||
reasons: list[str] = []
|
||||
recorded_type = recorded.get("sensor_type") or ""
|
||||
if recorded_type and sensor_type and recorded_type != sensor_type:
|
||||
reasons.append(f"sensor_type: {recorded_type} != {sensor_type}")
|
||||
recorded_config = recorded.get("config_ref") or ""
|
||||
if recorded_config and not self._same_path(recorded_config, config_ref):
|
||||
reasons.append(f"config_ref: {recorded_config} != {config_ref}")
|
||||
return reasons
|
||||
|
||||
def _on_calibration_apply(self, path: str) -> None:
|
||||
"""Push freshly exported coefficients into the device — on confirmation.
|
||||
|
||||
The user is shown the exact target device first, and only the matching
|
||||
worker is updated.
|
||||
worker is updated. A calibration that names a *different* device or
|
||||
configuration is called out explicitly and defaults to "no".
|
||||
"""
|
||||
path = str(path or self._last_calibration_export)
|
||||
if not path:
|
||||
@@ -931,6 +1016,29 @@ class StudioMainWindow(QMainWindow):
|
||||
return
|
||||
sensor_type = self.sensor_picker.sensor_type()
|
||||
config_ref = self._resolve_config_ref() or ""
|
||||
reasons = self.calibration_target_mismatch(
|
||||
path, sensor_type=sensor_type, config_ref=config_ref
|
||||
)
|
||||
if reasons:
|
||||
answer = QMessageBox.question(
|
||||
self,
|
||||
t("cal.apply_mismatch_title"),
|
||||
t(
|
||||
"cal.apply_mismatch",
|
||||
device=self.calibration_identity(path).get("sensor_type") or "?",
|
||||
detail="; ".join(reasons),
|
||||
sensor_type=sensor_type,
|
||||
config=config_ref,
|
||||
),
|
||||
QMessageBox.Yes | QMessageBox.No,
|
||||
QMessageBox.No,
|
||||
)
|
||||
if answer != QMessageBox.Yes:
|
||||
self.recorder.log_event(
|
||||
"calibration_apply_blocked", "; ".join(reasons)
|
||||
)
|
||||
self.statusBar().showMessage(t("cal.apply_mismatch_skipped"))
|
||||
return
|
||||
answer = QMessageBox.question(
|
||||
self,
|
||||
t("cal.apply_title"),
|
||||
@@ -952,30 +1060,32 @@ class StudioMainWindow(QMainWindow):
|
||||
t("cal.apply_pending", sensor_type=sensor_type)
|
||||
)
|
||||
|
||||
def _on_calibration_applied(self, path: str, version: str, frame_id: int) -> None:
|
||||
def _on_calibration_applied(
|
||||
self, path: str, version: str, frame_id: int, mono_t: float, session: int
|
||||
) -> None:
|
||||
"""Called from the acquisition thread once new coefficients are loaded."""
|
||||
# The acquisition thread must not touch Qt, so only plain state moves here.
|
||||
self._pending_calibration_events.append(
|
||||
(str(path), str(version), int(frame_id))
|
||||
)
|
||||
self.recording.note_calibration_applied(path, version, frame_id, mono_t, session)
|
||||
|
||||
def _on_reset_applied(self, frame_id: int, mono_t: float, session: int) -> None:
|
||||
"""Called from the acquisition thread once a zeroing took effect."""
|
||||
self.recording.note_reset(frame_id, mono_t, session)
|
||||
|
||||
def _drain_calibration_events(self) -> None:
|
||||
"""Publish confirmed calibration changes on the UI thread."""
|
||||
while self._pending_calibration_events:
|
||||
path, version, frame_id = self._pending_calibration_events.pop(0)
|
||||
# The version is the calibration's content hash: a path can be
|
||||
# re-exported with different coefficients, a hash cannot.
|
||||
self._active_calibration_id = version or path
|
||||
self.session_store.set_calibration_id(self._active_calibration_id)
|
||||
self.recorder.set_calibration_version(self._active_calibration_id)
|
||||
self.recorder.log_event(
|
||||
"calibration_applied",
|
||||
f"{path} ({self._active_calibration_id[:12] or 'unhashed'})",
|
||||
frame_id=frame_id,
|
||||
)
|
||||
self.statusBar().showMessage(
|
||||
t("cal.applied", version=self._active_calibration_id[:12])
|
||||
)
|
||||
for event in self.recording.drain_calibration_events():
|
||||
version = event["version"]
|
||||
if version:
|
||||
self.statusBar().showMessage(t("cal.applied", version=version[:12]))
|
||||
else:
|
||||
# An empty version is the honest answer for a device with no
|
||||
# calibration; it must not read as "the last one is still on".
|
||||
self.statusBar().showMessage(t("cal.applied_none"))
|
||||
|
||||
def _drain_reset_events(self) -> None:
|
||||
"""Publish confirmed zeroings on the UI thread."""
|
||||
for _event in self.recording.drain_reset_events():
|
||||
self.statusBar().showMessage(t("status.reset_ok"))
|
||||
|
||||
def _worker_for(self, sensor_type: str):
|
||||
self._ensure_workers()
|
||||
@@ -1228,7 +1338,6 @@ class StudioMainWindow(QMainWindow):
|
||||
# starts empty and is filled from that confirmation — a path is not a
|
||||
# version, and the same path can hold different numbers.
|
||||
self._active_calibration_id = ""
|
||||
self._pending_calibration_events.clear()
|
||||
self._record_error_count = 0
|
||||
self._dropped_reported = 0
|
||||
self._depth_range_value = None
|
||||
@@ -1318,18 +1427,12 @@ class StudioMainWindow(QMainWindow):
|
||||
if self._session_phase != "running" or self._active_worker is None:
|
||||
return
|
||||
try:
|
||||
# A request, not the zeroing itself: the acquisition thread applies it
|
||||
# between two frames and reports the frame and time it took effect.
|
||||
self._active_worker.reset()
|
||||
self.force_plots.reset()
|
||||
self._last_plot_packet_key = None
|
||||
# Recorded with the frame it happened at: a reset clears the SDK's
|
||||
# tracker, so an offline depth replay can start here instead of
|
||||
# guessing at the algorithm's earlier state.
|
||||
self.recorder.log_event(
|
||||
"zeroing",
|
||||
"tracker reset",
|
||||
frame_id=self._current_frame_id(),
|
||||
)
|
||||
self.statusBar().showMessage(t("status.reset_ok"))
|
||||
self.statusBar().showMessage(t("status.reset_pending"))
|
||||
except Exception as exc:
|
||||
self.statusBar().showMessage(t("status.run_failed", error=str(exc)))
|
||||
|
||||
@@ -1351,6 +1454,11 @@ class StudioMainWindow(QMainWindow):
|
||||
def _on_run_tick(self) -> None:
|
||||
if self._active_worker is None:
|
||||
return
|
||||
# Record bookkeeping happens before anything that can return early: a
|
||||
# recording that failed while the device was quiet still has to update
|
||||
# the button and the summary.
|
||||
self._drain_record_alarms()
|
||||
self._poll_record_report()
|
||||
if self._last_session_error:
|
||||
err = self._last_session_error
|
||||
self._last_session_error = ""
|
||||
@@ -1405,31 +1513,31 @@ class StudioMainWindow(QMainWindow):
|
||||
|
||||
if is_new:
|
||||
self._last_plot_packet_key = packet_key
|
||||
# Invalid readouts must not be drawn as if they were data, and they
|
||||
# must not be written into the plot buffer: NaN would poison the
|
||||
# auto-range and a zero would be indistinguishable from a real one.
|
||||
if packet.valid:
|
||||
# The live view consumes the *same* standardised measurement as the
|
||||
# CSV and the session store: one conversion decides what a value
|
||||
# means, so the chart cannot show something the record does not
|
||||
# contain (or vice versa).
|
||||
record = to_record_sample(packet)
|
||||
if record.valid:
|
||||
display = select_display_forces(
|
||||
units_pref=self.toolbar.units(),
|
||||
force_calibrated=packet.force_calibrated,
|
||||
fn=packet.normal_force,
|
||||
fx=packet.shear_x,
|
||||
fy=packet.shear_y,
|
||||
fn_raw=packet.normal_force_raw,
|
||||
fx_raw=packet.shear_x_raw,
|
||||
fy_raw=packet.shear_y_raw,
|
||||
sample=record,
|
||||
)
|
||||
if display.using_raw_fallback:
|
||||
self.statusBar().showMessage(t("status.units_raw_fallback"))
|
||||
self.force_plots.push(
|
||||
packet.timestamp,
|
||||
display.fn,
|
||||
display.fx,
|
||||
display.fy,
|
||||
# Session-relative time, the same base the store and
|
||||
# ``forces.csv`` use. The wall clock must never reach the
|
||||
# buffer: it would put the live axis on a different scale.
|
||||
self._session_time(packet),
|
||||
display.values[0],
|
||||
display.values[1],
|
||||
display.values[2],
|
||||
unit=display.unit,
|
||||
available=display.available,
|
||||
)
|
||||
else:
|
||||
self._report_invalid_sample(packet.invalid_reason)
|
||||
self._report_invalid_sample(record.invalid_reason)
|
||||
# The calibration panel is a *view*: sampling is driven by the
|
||||
# record channel, so hiding the panel can no longer stall it.
|
||||
self.cal_panel.set_live(
|
||||
@@ -1441,8 +1549,8 @@ class StudioMainWindow(QMainWindow):
|
||||
# Finalise calibration sampling on the UI thread once its deadline passes.
|
||||
self.cal_recorder.tick()
|
||||
self.cal_panel.poll_recorder()
|
||||
self._drain_record_alarms()
|
||||
self._drain_calibration_events()
|
||||
self._drain_reset_events()
|
||||
self._report_dropped_samples()
|
||||
|
||||
mode = self.arrow_panel.view_mode()
|
||||
@@ -1451,6 +1559,23 @@ class StudioMainWindow(QMainWindow):
|
||||
if self._show_fps and is_new:
|
||||
self.arrow_panel.set_fps(packet.fps)
|
||||
|
||||
def _session_time(self, packet) -> float:
|
||||
"""Session-relative acquisition time for the live view.
|
||||
|
||||
The session store already defines the origin (its first appended sample),
|
||||
and ``forces.csv`` uses the same base. Deriving the live time from that
|
||||
one origin is what keeps the live buffer, a rebuilt history and the replay
|
||||
on a single X axis.
|
||||
"""
|
||||
origin = self.session_store.origin_time()
|
||||
absolute = getattr(packet, "mono_t", None)
|
||||
if absolute is None:
|
||||
absolute = getattr(packet, "timestamp", 0.0)
|
||||
try:
|
||||
return float(absolute) - float(origin)
|
||||
except (TypeError, ValueError):
|
||||
return 0.0
|
||||
|
||||
def _report_invalid_sample(self, reason: str) -> None:
|
||||
"""Surface failed readouts instead of letting them pass as data.
|
||||
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Calibration service: the panel's operations, without the panel.
|
||||
|
||||
The panel shows a table and emits status messages; everything that touches the
|
||||
file system or a device identity happens here:
|
||||
|
||||
* preparing the session once (store, diagnostics and export share one model);
|
||||
* saving the audit record, reopening one, and exporting the coefficients;
|
||||
* keeping a reopened record's **own** device identity instead of restamping it
|
||||
with whatever device happens to be connected.
|
||||
|
||||
This is deliberately Qt-free so it can be tested without a display.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from orisys_app_shared.force.calibration_io import (
|
||||
CHANNEL_KEYS,
|
||||
CalSessionStore,
|
||||
export_force_calibration_v2,
|
||||
meta_from_sensor_json,
|
||||
)
|
||||
from orisys_app_shared.force.calibration_session import (
|
||||
CalibrationSessionRecord,
|
||||
read_calibration_session,
|
||||
session_filename,
|
||||
write_calibration_session,
|
||||
)
|
||||
from orisys_app_shared.paths import calibration_export_dir
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CalibrationService:
|
||||
"""Owns the calibration session's files and its prepared model."""
|
||||
|
||||
def __init__(self, *, store: Optional[CalSessionStore] = None) -> None:
|
||||
self.store = store or CalSessionStore()
|
||||
#: The one prepared model the panel displays and exports.
|
||||
self.prepared: Any = None
|
||||
#: Device identity of a reopened record, while it is open.
|
||||
self.record_identity: Optional[dict] = None
|
||||
#: The record currently open, if any.
|
||||
self.open_record: Optional[CalibrationSessionRecord] = None
|
||||
|
||||
# --------------------------------------------------------------- identity
|
||||
def export_meta(
|
||||
self,
|
||||
*,
|
||||
sensor_type: str,
|
||||
config_ref: str | None,
|
||||
sn: str = "",
|
||||
duration_s: float = 0.0,
|
||||
) -> dict:
|
||||
"""The identity/pipeline metadata an export would carry.
|
||||
|
||||
While a saved record is open its **own** device identity wins: reopening
|
||||
device A's calibration on a machine connected to device B must keep saying
|
||||
A, or the evidence is silently restamped with B's serial number.
|
||||
"""
|
||||
meta = meta_from_sensor_json(config_ref)
|
||||
meta["sensor_type"] = str(sensor_type or "box")
|
||||
meta["sn"] = str(sn or "")
|
||||
meta["record_duration_s"] = float(duration_s or 0.0)
|
||||
identity = self.record_identity
|
||||
if identity:
|
||||
meta["sensor_type"] = identity.get("sensor_type") or meta["sensor_type"]
|
||||
meta["sn"] = identity.get("sn") or meta["sn"]
|
||||
meta["config_ref"] = identity.get("config_ref") or meta.get("config_ref")
|
||||
meta["source_record"] = str(identity.get("path") or "")
|
||||
meta["source_created_at"] = str(identity.get("created_at") or "")
|
||||
return meta
|
||||
|
||||
def clear_record_identity(self) -> None:
|
||||
"""A new run is the live device's context again."""
|
||||
self.record_identity = None
|
||||
self.open_record = None
|
||||
|
||||
# ------------------------------------------------------------ preparation
|
||||
def prepare(self, recorder, meta: dict) -> Any:
|
||||
"""Prepare the session once: store, diagnostics and export share it."""
|
||||
self.prepared = None
|
||||
self.store.clear()
|
||||
if recorder is None:
|
||||
return None
|
||||
prepared = recorder.prepared(meta=meta)
|
||||
self.prepared = prepared
|
||||
for key in CHANNEL_KEYS:
|
||||
self.store.set_channel(key, prepared.channel_samples(key))
|
||||
# The residuals belong to the points, because the table shows them.
|
||||
for point in recorder.all_points():
|
||||
value = prepared.residuals.get(id(point))
|
||||
point.residual_n = float("nan") if value is None else float(value)
|
||||
return prepared
|
||||
|
||||
# ------------------------------------------------------------------- files
|
||||
def save_record(self, recorder, meta: dict) -> Path:
|
||||
"""Write the audit record. Returns the written path."""
|
||||
points = recorder.all_points()
|
||||
path = calibration_export_dir() / session_filename(meta.get("sensor_type"))
|
||||
return write_calibration_session(path, points, meta=meta)
|
||||
|
||||
def load_record(self, path: str | Path) -> CalibrationSessionRecord:
|
||||
"""Reopen a saved session and remember whose record it is."""
|
||||
record = read_calibration_session(path)
|
||||
identity = record.identity()
|
||||
self.record_identity = {**identity, "path": str(record.path)}
|
||||
self.open_record = record
|
||||
return record
|
||||
|
||||
def export_coefficients(
|
||||
self,
|
||||
*,
|
||||
sensor_type: str,
|
||||
export_dir: str | Path | None = None,
|
||||
local_bundle_dir: str | Path | None = None,
|
||||
) -> tuple[Path, Path | None, list[str]]:
|
||||
"""Serialise the prepared model. Raises ValueError when there is none."""
|
||||
if self.prepared is None or not getattr(self.prepared, "ok", False):
|
||||
warnings = list(getattr(self.prepared, "warnings", []) or [])
|
||||
raise ValueError("; ".join(warnings) or "no complete channels")
|
||||
return export_force_calibration_v2(
|
||||
self.prepared,
|
||||
export_dir=export_dir or calibration_export_dir(),
|
||||
sensor_type=str(sensor_type or "box"),
|
||||
local_bundle_dir=local_bundle_dir,
|
||||
)
|
||||
@@ -0,0 +1,266 @@
|
||||
"""The record channel: one experiment directory, one session store, one sampler.
|
||||
|
||||
This is the coordination the main window used to do inline — and it is deliberately
|
||||
free of Qt. Three consumers are fed from the same acquisition packet:
|
||||
|
||||
* the :class:`ExperimentRecorder`, which writes the directory;
|
||||
* the whole-session store the charts and the browser read;
|
||||
* the calibration sampler (``CalibrationRecorder``), whose lifecycle must not
|
||||
depend on the calibration panel being visible.
|
||||
|
||||
The window keeps the buttons, the menus and the status messages; the decisions
|
||||
about *what* is recorded, *how* it is named and *what actually happened* live
|
||||
here, where they can be tested without a display.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig, RecordReport
|
||||
from orisys_app_shared.record.calibration_snapshot import store_calibration_snapshot
|
||||
from orisys_app_shared.record.sample import to_record_sample
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RecordingCoordinator:
|
||||
"""Owns the recorder and the per-packet fan-out."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
session_store,
|
||||
cal_recorder,
|
||||
recorder: Optional[ExperimentRecorder] = None,
|
||||
) -> None:
|
||||
self.session_store = session_store
|
||||
self.cal_recorder = cal_recorder
|
||||
self.recorder = recorder or ExperimentRecorder()
|
||||
#: Confirmed calibration changes: ``(path, version, frame_id, mono_t, session)``.
|
||||
self._calibration_events: list[tuple[str, str, int, float, int]] = []
|
||||
#: Confirmed zeroings: ``(frame_id, mono_t, session)``.
|
||||
self._reset_events: list[tuple[int, float, int]] = []
|
||||
#: Raised by the writer thread; drained on the UI thread.
|
||||
self._alarms: list[str] = []
|
||||
#: A stop was requested and the writer has not closed its files yet.
|
||||
self.report_pending = False
|
||||
self.last_report: Optional[RecordReport] = None
|
||||
self._active_calibration_id = ""
|
||||
self._record_fps = 30.0
|
||||
self.recorder.set_alarm_handler(self._on_alarm)
|
||||
|
||||
# ------------------------------------------------------------- acquisition
|
||||
def on_packet(self, packet):
|
||||
"""Runs on the **acquisition thread**. Must stay cheap and Qt-free.
|
||||
|
||||
The packet is converted exactly once, by the same function the CSV writer
|
||||
uses, so the store and the file can never disagree about whether a value
|
||||
is a measurement. Returns the converted record, which the caller may use
|
||||
for its own (UI-side) bookkeeping.
|
||||
"""
|
||||
record = to_record_sample(
|
||||
packet, calibration_version=self._active_calibration_id
|
||||
)
|
||||
from orisys_app_shared.force.session import SessionSample
|
||||
|
||||
sample = SessionSample.from_record(record)
|
||||
self.session_store.append(sample)
|
||||
self.cal_recorder.feed(sample)
|
||||
# Video + CSV + metadata share this one record path.
|
||||
self.recorder.on_packet(packet)
|
||||
fps = float(getattr(packet, "fps", 0.0) or 0.0)
|
||||
if fps > 0.0:
|
||||
self._record_fps = fps
|
||||
return record
|
||||
|
||||
def take_preroll(self, worker) -> list:
|
||||
"""Frame history from the active worker, if it keeps one."""
|
||||
take = getattr(worker, "take_preroll", None)
|
||||
if not callable(take):
|
||||
return []
|
||||
try:
|
||||
return list(take())
|
||||
except Exception: # noqa: BLE001 - the history is optional, not the record
|
||||
logger.exception("Could not take the pre-roll buffer")
|
||||
return []
|
||||
|
||||
def set_active_calibration(self, version: str) -> None:
|
||||
"""The calibration the record should name until the device says otherwise."""
|
||||
self._active_calibration_id = str(version or "")
|
||||
|
||||
@property
|
||||
def active_calibration(self) -> str:
|
||||
return self._active_calibration_id
|
||||
|
||||
@property
|
||||
def record_fps(self) -> float:
|
||||
return self._record_fps
|
||||
|
||||
# ------------------------------------------------------------------ events
|
||||
def note_calibration_applied(
|
||||
self, path: str, version: str, frame_id: int, mono_t: float, session: int
|
||||
) -> None:
|
||||
"""Called from the acquisition thread once coefficients are loaded."""
|
||||
self._calibration_events.append(
|
||||
(str(path), str(version), int(frame_id), float(mono_t), int(session))
|
||||
)
|
||||
|
||||
def note_reset(self, frame_id: int, mono_t: float, session: int) -> None:
|
||||
"""Called from the acquisition thread once a zeroing took effect."""
|
||||
self._reset_events.append((int(frame_id), float(mono_t), int(session)))
|
||||
|
||||
def drain_calibration_events(self) -> list[dict]:
|
||||
"""Publish confirmed calibrations. Returns what the UI should show."""
|
||||
published: list[dict] = []
|
||||
while self._calibration_events:
|
||||
path, version, frame_id, mono_t, session = self._calibration_events.pop(0)
|
||||
# The version is the calibration's content hash: a path can be
|
||||
# re-exported with different coefficients, a hash cannot.
|
||||
self._active_calibration_id = version
|
||||
self.session_store.set_calibration_id(version)
|
||||
self.recorder.set_calibration_version(version)
|
||||
snapshot = self.snapshot_calibration(path, version, mono_t, frame_id)
|
||||
self.recorder.log_event(
|
||||
"calibration_applied",
|
||||
f"{path} ({version[:12] or 'unhashed'})"
|
||||
+ (f" -> {snapshot}" if snapshot else ""),
|
||||
mono_t=mono_t,
|
||||
frame_id=frame_id,
|
||||
)
|
||||
published.append(
|
||||
{
|
||||
"path": path,
|
||||
"version": version,
|
||||
"frame_id": frame_id,
|
||||
"mono_t": mono_t,
|
||||
"session": session,
|
||||
"snapshot": snapshot,
|
||||
}
|
||||
)
|
||||
return published
|
||||
|
||||
def drain_reset_events(self) -> list[dict]:
|
||||
"""Publish confirmed zeroings."""
|
||||
published: list[dict] = []
|
||||
while self._reset_events:
|
||||
frame_id, mono_t, session = self._reset_events.pop(0)
|
||||
self.recorder.log_event(
|
||||
"zeroing",
|
||||
"tracker reset (applied by the acquisition thread)",
|
||||
mono_t=mono_t,
|
||||
frame_id=frame_id,
|
||||
)
|
||||
published.append(
|
||||
{"frame_id": frame_id, "mono_t": mono_t, "session": session}
|
||||
)
|
||||
return published
|
||||
|
||||
def snapshot_calibration(
|
||||
self, path: str, version: str, mono_t: float, frame_id: int
|
||||
) -> str:
|
||||
"""Copy the applied coefficients into the recording. Returns its name."""
|
||||
directory = self.recorder.directory
|
||||
if directory is None or not path or not Path(path).is_file():
|
||||
return ""
|
||||
try:
|
||||
name = store_calibration_snapshot(directory, Path(path), version)
|
||||
except Exception: # noqa: BLE001 - the record continues without it
|
||||
logger.exception("Could not snapshot the applied calibration")
|
||||
return ""
|
||||
self.recorder.note_calibration_snapshot(
|
||||
version=version, name=name, frame_id=frame_id, mono_t=mono_t
|
||||
)
|
||||
return name
|
||||
|
||||
# ------------------------------------------------------------------ alarms
|
||||
def _on_alarm(self, message: str) -> None:
|
||||
"""Called from the writer thread. Queue it; never touch Qt here."""
|
||||
logger.error("Recording alarm: %s", message)
|
||||
self._alarms.append(str(message))
|
||||
|
||||
def drain_alarms(self) -> list[str]:
|
||||
drained, self._alarms = self._alarms, []
|
||||
return drained
|
||||
|
||||
# --------------------------------------------------------------- lifecycle
|
||||
def start(
|
||||
self,
|
||||
*,
|
||||
root: Path,
|
||||
config: RecordConfig,
|
||||
metadata: dict[str, Any],
|
||||
calibration_id: str = "",
|
||||
calibration_version: str = "",
|
||||
preroll=(),
|
||||
) -> Path:
|
||||
"""Start a recording. Raises when the directory cannot be created."""
|
||||
directory = self.recorder.start(
|
||||
root=root,
|
||||
session_id=int(self.session_store.session_id),
|
||||
metadata=metadata,
|
||||
calibration_id=calibration_id,
|
||||
calibration_version=calibration_version,
|
||||
preroll=preroll,
|
||||
)
|
||||
self.last_report = None
|
||||
self.report_pending = False
|
||||
return Path(directory)
|
||||
|
||||
def stop(self, *, wait: bool = True) -> Optional[RecordReport]:
|
||||
"""Stop and return the report, or ``None`` when nothing was started."""
|
||||
if self.recorder.directory is None and self.recorder.report is None:
|
||||
self.report_pending = False
|
||||
return None
|
||||
if self.recorder.is_recording:
|
||||
self.recorder.log_event("record_stopped", "")
|
||||
report = self.recorder.stop(wait=wait)
|
||||
self.publish(report)
|
||||
return report
|
||||
|
||||
def publish(self, report: Optional[RecordReport]) -> None:
|
||||
"""Record a report; a completed one becomes ``last_report``."""
|
||||
if report is None:
|
||||
return
|
||||
if not report.completed:
|
||||
# The writer is still closing; the UI timer publishes the real one.
|
||||
self.report_pending = True
|
||||
return
|
||||
self.report_pending = False
|
||||
self.last_report = report
|
||||
|
||||
def poll(self) -> Optional[RecordReport]:
|
||||
"""The terminal report, once the writer has actually closed the files."""
|
||||
if not self.report_pending:
|
||||
return None
|
||||
report = self.recorder.report
|
||||
if report is None:
|
||||
return None
|
||||
self.publish(report)
|
||||
return report
|
||||
|
||||
@property
|
||||
def is_recording(self) -> bool:
|
||||
return self.recorder.is_recording
|
||||
|
||||
@property
|
||||
def directory(self):
|
||||
return self.recorder.directory
|
||||
|
||||
|
||||
def record_config_from_prefs(
|
||||
prefs: dict,
|
||||
*,
|
||||
mode: str,
|
||||
fps: float,
|
||||
on_problem: Optional[Callable[[str], None]] = None,
|
||||
) -> RecordConfig:
|
||||
"""Build the record config the toolbar/prefs describe."""
|
||||
return RecordConfig(
|
||||
video=(mode == "video_data"),
|
||||
lossless_frames=bool(prefs.get("record_lossless_frames", False)),
|
||||
fps=float(fps),
|
||||
segment_seconds=float(prefs.get("record_segment_seconds", 300.0)),
|
||||
)
|
||||
@@ -30,18 +30,16 @@ from orisys_app_shared.force.calibration_io import (
|
||||
CalSample,
|
||||
CalSessionStore,
|
||||
ChannelKey,
|
||||
export_force_calibration_v2,
|
||||
meta_from_sensor_json,
|
||||
)
|
||||
from orisys_app_shared.force.calibration_session import (
|
||||
read_calibration_session,
|
||||
session_filename,
|
||||
write_calibration_session,
|
||||
)
|
||||
from orisys_app_shared.i18n import t
|
||||
from orisys_app_shared.paths import calibration_export_dir
|
||||
from orisys_app_shared.theme import color
|
||||
|
||||
from orisys_studio.calibration_service import CalibrationService
|
||||
from orisys_studio.ui.cal_point_dialog import CalibrationPointDialog
|
||||
|
||||
CHANNEL_LABELS = ("fn", "fx", "fy")
|
||||
@@ -71,7 +69,10 @@ class CalPanel(QFrame):
|
||||
super().__init__(parent)
|
||||
self.setObjectName("contentPanel")
|
||||
self._theme = theme
|
||||
self._store = CalSessionStore()
|
||||
#: The files, the identity and the prepared model live in the service;
|
||||
#: this panel shows them and emits status messages.
|
||||
self._service = CalibrationService()
|
||||
self._store = self._service.store
|
||||
self._channel: ChannelKey = "fn"
|
||||
self._running = False
|
||||
#: Set by :meth:`bind_recorder`. Sampling lives there, not here, so the
|
||||
@@ -244,6 +245,14 @@ class CalPanel(QFrame):
|
||||
self.retranslate()
|
||||
|
||||
# ------------------------------------------------------------------ API
|
||||
@property
|
||||
def _prepared(self):
|
||||
return self._service.prepared
|
||||
|
||||
@property
|
||||
def _record_identity(self):
|
||||
return self._service.record_identity
|
||||
|
||||
def set_theme(self, theme: str) -> None:
|
||||
self._theme = theme
|
||||
self._render_live_label()
|
||||
@@ -266,6 +275,8 @@ class CalPanel(QFrame):
|
||||
def set_meta_context(self, *, sensor_type: str, config_ref: str | None) -> None:
|
||||
self._sensor_type = str(sensor_type or "box")
|
||||
self._config_ref = config_ref
|
||||
# A new run is the live device's context again.
|
||||
self._service.clear_record_identity()
|
||||
|
||||
def set_live(self, fn: float, fx: float, fy: float) -> None:
|
||||
"""Update the live readout. Sampling is handled by the recorder."""
|
||||
@@ -519,7 +530,8 @@ class CalPanel(QFrame):
|
||||
if self._recorder is None:
|
||||
self.fit_label.setText("")
|
||||
return
|
||||
fit = self._recorder.fit(self._channel)
|
||||
# Same prepared result the store and the export use — never a second one.
|
||||
fit = self._recorder.fit(self._channel, prepared=self._prepared)
|
||||
if not fit.usable:
|
||||
self.fit_label.setText(t("cal.fit_insufficient", count=fit.n_points))
|
||||
return
|
||||
@@ -549,12 +561,30 @@ class CalPanel(QFrame):
|
||||
state = t("cal.state_restore") if point.excluded else t("cal.state_exclude")
|
||||
self.btn_toggle_excluded.setText(state)
|
||||
|
||||
def point_dialog(self, point):
|
||||
"""The diagnostic view for a point, rebuilt from its samples if needed.
|
||||
|
||||
A reopened record carries the full sample series, so the three-axis view
|
||||
can always be produced — waiting for a cached trace would have made
|
||||
exactly the reopened points (the ones worth auditing) unopenable.
|
||||
"""
|
||||
if point is None or not (point.trace or point.samples):
|
||||
return None
|
||||
if not point.trace:
|
||||
from orisys_app_shared.force.calibration_session import (
|
||||
restore_diagnostics,
|
||||
)
|
||||
|
||||
restore_diagnostics(point)
|
||||
if not point.trace:
|
||||
return None
|
||||
return CalibrationPointDialog(point, theme=self._theme, parent=self)
|
||||
|
||||
def _on_row_activated(self) -> None:
|
||||
"""Double click a row to inspect that point's three-axis curve."""
|
||||
point = self._current_point()
|
||||
if point is None or not point.trace:
|
||||
return
|
||||
CalibrationPointDialog(point, theme=self._theme, parent=self).exec_()
|
||||
dialog = self.point_dialog(self._current_point())
|
||||
if dialog is not None:
|
||||
dialog.exec_()
|
||||
|
||||
def _on_toggle_excluded(self) -> None:
|
||||
point = self._current_point()
|
||||
@@ -580,33 +610,25 @@ class CalPanel(QFrame):
|
||||
t("cal.marked_validation" if point.is_validation else "cal.unmarked_validation")
|
||||
)
|
||||
|
||||
def _rebuild_store(self) -> None:
|
||||
"""Rebuild the export store from the points that are still in the fit.
|
||||
|
||||
Validation points are held out: they exist to check the model, so letting
|
||||
them into the build would both corrupt it and make the "independent"
|
||||
error meaningless. Repeated reference loads are merged rather than
|
||||
overwritten, so every sample that was actually taken reaches the model.
|
||||
"""
|
||||
self._store.clear()
|
||||
if self._recorder is None:
|
||||
return
|
||||
for point in self._recorder.all_points():
|
||||
if point.excluded or point.is_validation:
|
||||
continue
|
||||
stats = point.loaded_stats()
|
||||
if not np.isfinite(stats.mean):
|
||||
continue
|
||||
self._store.aggregate(
|
||||
point.channel,
|
||||
CalSample(
|
||||
weight_n=float(point.reference_load_n),
|
||||
raw_mean=float(stats.mean),
|
||||
n=int(stats.n),
|
||||
std=float(stats.std) if np.isfinite(stats.std) else 0.0,
|
||||
),
|
||||
def _export_meta(self) -> dict:
|
||||
"""The identity/pipeline metadata the export would carry."""
|
||||
return self._service.export_meta(
|
||||
sensor_type=self._sensor_type,
|
||||
config_ref=self._config_ref,
|
||||
sn=self.sn_edit.text().strip(),
|
||||
duration_s=float(self.duration_spin.value()),
|
||||
)
|
||||
|
||||
def _rebuild_store(self) -> None:
|
||||
"""Prepare the session once: store, diagnostics and export share it.
|
||||
|
||||
The panel used to filter, aggregate and build its own model while
|
||||
``CalibrationRecorder.fit`` prepared a second one for the table, so the
|
||||
displayed residual and the exported curve could describe different
|
||||
models. Both now read :class:`PreparedCalibration`, built by the service.
|
||||
"""
|
||||
self._service.prepare(self._recorder, self._export_meta())
|
||||
|
||||
def _on_delete_row(self) -> None:
|
||||
point = self._current_point()
|
||||
if point is None or self._recorder is None:
|
||||
@@ -630,20 +652,24 @@ class CalPanel(QFrame):
|
||||
self._refresh_table()
|
||||
|
||||
def _on_export(self) -> None:
|
||||
meta = meta_from_sensor_json(self._config_ref)
|
||||
meta["sensor_type"] = self._sensor_type
|
||||
meta["sn"] = self.sn_edit.text().strip()
|
||||
meta["record_duration_s"] = float(self.duration_spin.value())
|
||||
# Re-prepare first: the SN and the requested duration may have changed,
|
||||
# and the file must be exactly the model the panel is showing.
|
||||
self._rebuild_store()
|
||||
self._refresh_fit_label()
|
||||
if self._prepared is None or not self._prepared.ok:
|
||||
QMessageBox.warning(
|
||||
self,
|
||||
t("panels.cal"),
|
||||
t("cal.export_no_model", detail="; ".join(self._warnings())),
|
||||
)
|
||||
return
|
||||
try:
|
||||
# Write into the user's calibration directory only. Overwriting the
|
||||
# device bundle under config/sensors/<type>/ is now an explicit,
|
||||
# separate step so that exporting cannot silently change what the
|
||||
# next Sensor() will load.
|
||||
primary, _local_path, warnings = export_force_calibration_v2(
|
||||
self._store,
|
||||
export_dir=calibration_export_dir(),
|
||||
primary, _local_path, warnings = self._service.export_coefficients(
|
||||
sensor_type=self._sensor_type,
|
||||
meta=meta,
|
||||
local_bundle_dir=None,
|
||||
)
|
||||
except ValueError as exc:
|
||||
@@ -664,6 +690,9 @@ class CalPanel(QFrame):
|
||||
self.calibration_exported.emit(str(primary))
|
||||
QMessageBox.information(self, t("panels.cal"), msg)
|
||||
|
||||
def _warnings(self) -> list[str]:
|
||||
return list(getattr(self._prepared, "warnings", []) or [])
|
||||
|
||||
def _on_apply(self) -> None:
|
||||
if not self._last_export_path:
|
||||
self.status_message.emit(t("cal.apply_no_export"))
|
||||
@@ -678,12 +707,10 @@ class CalPanel(QFrame):
|
||||
if not points:
|
||||
self.status_message.emit(t("cal.save_record_empty"))
|
||||
return
|
||||
meta = meta_from_sensor_json(self._config_ref)
|
||||
meta["sensor_type"] = self._sensor_type
|
||||
meta["sn"] = self.sn_edit.text().strip()
|
||||
path = calibration_export_dir() / session_filename(self._sensor_type)
|
||||
# ``_export_meta`` keeps a reopened record's own device identity.
|
||||
meta = self._export_meta()
|
||||
try:
|
||||
written = write_calibration_session(path, points, meta=meta)
|
||||
written = self._service.save_record(self._recorder, meta)
|
||||
except Exception as exc: # noqa: BLE001 - reported to the operator
|
||||
QMessageBox.warning(
|
||||
self, t("panels.cal"), t("cal.save_record_failed", error=str(exc))
|
||||
@@ -694,10 +721,11 @@ class CalPanel(QFrame):
|
||||
self.status_message.emit(msg)
|
||||
QMessageBox.information(self, t("panels.cal"), msg)
|
||||
|
||||
def _on_open_record(self) -> None:
|
||||
"""Reopen a saved session so it can be inspected again."""
|
||||
def _on_open_record(self, path: str | None = None) -> None:
|
||||
"""Reopen a saved session: its points **and** its device identity."""
|
||||
if self._recorder is None:
|
||||
return
|
||||
if path is None:
|
||||
path, _filter = QFileDialog.getOpenFileName(
|
||||
self,
|
||||
t("cal.open_record"),
|
||||
@@ -707,17 +735,22 @@ class CalPanel(QFrame):
|
||||
if not path:
|
||||
return
|
||||
try:
|
||||
record = read_calibration_session(path)
|
||||
record = self._service.load_record(path)
|
||||
except Exception as exc: # noqa: BLE001 - reported to the operator
|
||||
QMessageBox.warning(
|
||||
self, t("panels.cal"), t("cal.open_record_failed", error=str(exc))
|
||||
)
|
||||
return
|
||||
# Show the record's own SN: it is what a later save or export will carry,
|
||||
# so the operator must not be looking at the connected device's number.
|
||||
if record.identity().get("sn"):
|
||||
self.sn_edit.setText(record.identity()["sn"])
|
||||
self._recorder.load_points(record.points, replace=True)
|
||||
self._rebuild_store()
|
||||
self._refresh_table()
|
||||
self._refresh_enabled()
|
||||
msg = t("cal.open_record_ok", path=record.path.name, points=len(record.points))
|
||||
msg += " " + t("cal.open_record_device", device=record.describe())
|
||||
if record.warnings:
|
||||
msg += "\n" + t("cal.open_record_warnings", detail="; ".join(record.warnings))
|
||||
self.status_message.emit(msg)
|
||||
|
||||
@@ -18,6 +18,7 @@ from PyQt5.QtWidgets import (
|
||||
from orisys_app_shared.acquisition.constants import FORCE_SAMPLE_RATE_HZ, LOWPASS_CUTOFF_HZ
|
||||
from orisys_app_shared.force import MetricHistory
|
||||
from orisys_app_shared.force.history import DEFAULT_MAX_FPS
|
||||
from orisys_app_shared.force.source import StoreCurveSource
|
||||
from orisys_app_shared.i18n import t
|
||||
from orisys_app_shared.theme import color
|
||||
|
||||
@@ -100,8 +101,13 @@ class ForcePlotsPanel(QFrame):
|
||||
self._fixed_newton = True
|
||||
self._unit = "raw"
|
||||
self._last_values: tuple[float, float, float] = (0.0, 0.0, 0.0)
|
||||
#: Per axis: does ``_last_values`` hold a real value in the active unit?
|
||||
self._available: tuple[bool, bool, bool] = (True, True, True)
|
||||
# -- browsing state -------------------------------------------------
|
||||
self._store = None
|
||||
#: Where the curves read from: the in-memory session store here, a
|
||||
#: recording on disk in the replay. One interface either way.
|
||||
self._source = StoreCurveSource()
|
||||
self._mode = "follow"
|
||||
self._window: tuple[float, float] = (0.0, 0.0)
|
||||
self._cursor_t = 0.0
|
||||
@@ -313,6 +319,7 @@ class ForcePlotsPanel(QFrame):
|
||||
self.history.reset()
|
||||
self._clear_curves()
|
||||
self._last_values = (0.0, 0.0, 0.0)
|
||||
self._available = (True, True, True)
|
||||
self._unit = "raw"
|
||||
self._mode = "follow"
|
||||
self._window = (0.0, 0.0)
|
||||
@@ -328,19 +335,33 @@ class ForcePlotsPanel(QFrame):
|
||||
|
||||
def push(
|
||||
self,
|
||||
timestamp: float,
|
||||
t_rel_s: float,
|
||||
fn: float,
|
||||
fx: float,
|
||||
fy: float,
|
||||
*,
|
||||
unit: str = "raw",
|
||||
available: tuple[bool, bool, bool] | None = None,
|
||||
) -> None:
|
||||
"""Append one live sample.
|
||||
|
||||
``t_rel_s`` is **session-relative** time — the same base the session store
|
||||
and ``forces.csv`` use. Handing the wall clock in here used to put the live
|
||||
buffer on a different time axis from a rebuilt history, so switching units
|
||||
(or entering browse) made the X axis jump by the size of the epoch.
|
||||
|
||||
``available`` marks axes that have no value in ``unit`` (a device with
|
||||
only its normal-force axis calibrated). Those keep NaN and show as a gap
|
||||
and a dash, never as a raw count wearing a newton label.
|
||||
"""
|
||||
if unit != self._unit:
|
||||
# Never mix units in one buffer. Callers switch units through
|
||||
# ``set_units``/``set_fixed_newton`` which rebuild from the session
|
||||
# store; a bare change here would silently reinterpret old samples.
|
||||
self._unit = unit
|
||||
self._last_values = (float(fn), float(fx), float(fy))
|
||||
if available is not None:
|
||||
self._available = tuple(bool(flag) for flag in available)
|
||||
self._render_value_cards()
|
||||
if self._mode == "browse":
|
||||
# Browsing is a display pause, not a capture pause: the record
|
||||
@@ -351,7 +372,7 @@ class ForcePlotsPanel(QFrame):
|
||||
# Store the true value. Clamping here (the old ``max(0.0, fn)``) erased
|
||||
# negative FN from the buffer and hid zero drift; out-of-range values are
|
||||
# now handled by the axis range, which only affects what is drawn.
|
||||
self.history.push(timestamp, float(fn), float(fx), float(fy))
|
||||
self.history.push(t_rel_s, float(fn), float(fx), float(fy))
|
||||
state = self.history.state()
|
||||
if state is None:
|
||||
return
|
||||
@@ -372,7 +393,9 @@ class ForcePlotsPanel(QFrame):
|
||||
self.plots[i].setYRange(*state.y_ranges[i], padding=0.0)
|
||||
finally:
|
||||
self._syncing = False
|
||||
self.set_cursor(float(timestamp) - (self._time_origin() or 0.0))
|
||||
# The buffer already carries session-relative time, so the cursor is the
|
||||
# time itself: subtracting an origin here would double-count it.
|
||||
self.set_cursor(float(t_rel_s))
|
||||
# The overview is a full-store read, so refresh it on a timer rather
|
||||
# than on every frame.
|
||||
now = time.perf_counter()
|
||||
@@ -381,12 +404,6 @@ class ForcePlotsPanel(QFrame):
|
||||
self._last_overview_at = now
|
||||
self._sync_overview()
|
||||
|
||||
def _time_origin(self) -> float:
|
||||
"""The store's time origin, so the cursor matches the overview axis."""
|
||||
if self._store is None:
|
||||
return 0.0
|
||||
return self._store.origin_time()
|
||||
|
||||
def set_units(self, unit: str, store=None) -> None:
|
||||
"""Switch display units without mixing units in the curve buffer.
|
||||
|
||||
@@ -398,11 +415,13 @@ class ForcePlotsPanel(QFrame):
|
||||
if unit == self._unit and store is None:
|
||||
return
|
||||
self._unit = unit
|
||||
# ``clear`` keeps the buffer's time origin on purpose: the rebuilt history
|
||||
# is in the same session-relative base, so the X axis must not restart.
|
||||
self.history.clear()
|
||||
self._clear_curves()
|
||||
self._overview_dirty = True
|
||||
if store is not None:
|
||||
self._store = store
|
||||
self.set_store(store)
|
||||
if self._mode == "browse":
|
||||
self.refresh_from_store()
|
||||
else:
|
||||
@@ -414,6 +433,7 @@ class ForcePlotsPanel(QFrame):
|
||||
def set_store(self, store) -> None:
|
||||
"""Attach the whole-session store the browser reads from."""
|
||||
self._store = store
|
||||
self._source.set_store(store)
|
||||
self._overview_dirty = True
|
||||
self._refresh_mode_label()
|
||||
|
||||
@@ -437,19 +457,19 @@ class ForcePlotsPanel(QFrame):
|
||||
|
||||
def show_all(self) -> None:
|
||||
"""Overview: fit the entire session, in browse mode."""
|
||||
if self._store is None:
|
||||
if self._source is None:
|
||||
return
|
||||
first, last = self._store.extent()
|
||||
first, last = self._source.extent()
|
||||
if last <= first:
|
||||
return
|
||||
self._set_mode("browse")
|
||||
self.set_window(first, last)
|
||||
|
||||
def set_window(self, t0: float, t1: float) -> None:
|
||||
"""Show ``[t0, t1]`` from the whole-session store."""
|
||||
if self._store is None:
|
||||
"""Show ``[t0, t1]`` from the whole-session source."""
|
||||
if self._source is None:
|
||||
return
|
||||
first, last = self._store.extent()
|
||||
first, last = self._source.extent()
|
||||
if t1 <= t0:
|
||||
t1 = t0 + max(0.001, self.history.window_sec)
|
||||
t0 = max(first, float(t0))
|
||||
@@ -463,21 +483,23 @@ class ForcePlotsPanel(QFrame):
|
||||
if self._store is None:
|
||||
return
|
||||
t0, t1 = self._window
|
||||
slice_ = self._store.visible(
|
||||
# One shared rule for every view, decided **per axis**: valid and finite
|
||||
# for *that* axis. An all-axes rule hid a valid FN whenever the device had
|
||||
# no shear calibration, and a finiteness-only rule let a stale value
|
||||
# reappear after another view excluded it.
|
||||
window = self._source.window(
|
||||
t0, t1, max_points=self._max_draw_points, unit=self._unit
|
||||
)
|
||||
self._syncing = True
|
||||
try:
|
||||
if len(slice_) == 0:
|
||||
if len(window) == 0:
|
||||
for curve in self.curves:
|
||||
curve.setData([], [])
|
||||
else:
|
||||
# One shared rule for every view: valid *and* finite. A stale
|
||||
# number left by a failed compute step must not reappear here
|
||||
# after another view already excluded it.
|
||||
times, values, _mask = slice_.plottable(self._unit)
|
||||
for index, curve in enumerate(self.curves):
|
||||
curve.setData(times, values[:, index])
|
||||
times, values = window.plottable_axis(index)
|
||||
curve.setData(times, values)
|
||||
self._availability_from_window(window)
|
||||
for plot in self.plots:
|
||||
plot.setXRange(t0, t1, padding=0.0)
|
||||
self._apply_axis_mode()
|
||||
@@ -523,7 +545,7 @@ class ForcePlotsPanel(QFrame):
|
||||
x0, x1 = float(x_range[0]), float(x_range[1])
|
||||
if not (x1 > x0):
|
||||
return
|
||||
first, last = self._store.extent()
|
||||
first, last = self._source.extent()
|
||||
if last <= first:
|
||||
return
|
||||
# Keep the window inside the recorded session.
|
||||
@@ -554,7 +576,7 @@ class ForcePlotsPanel(QFrame):
|
||||
self.stats_label.setText("")
|
||||
return
|
||||
t0, t1 = self._window
|
||||
stats = self._store.statistics(t0, t1, unit=self._unit)
|
||||
stats = self._source.statistics(t0, t1, unit=self._unit)
|
||||
if stats.samples == 0:
|
||||
self.stats_label.setText(t("chart.selection_empty"))
|
||||
return
|
||||
@@ -581,19 +603,19 @@ class ForcePlotsPanel(QFrame):
|
||||
def _sync_overview(self) -> None:
|
||||
if self._store is None:
|
||||
return
|
||||
first, last = self._store.extent()
|
||||
first, last = self._source.extent()
|
||||
if last <= first:
|
||||
self.overview_curve.setData([], [])
|
||||
return
|
||||
# The overview is a full-store read, so only rebuild it when it can have
|
||||
# changed (window/unit/session) rather than on every pan.
|
||||
slice_ = self._store.visible(
|
||||
first, last, max_points=self._overview_points(), unit=self._unit
|
||||
# The overview is a whole-session read, so only rebuild it when it can
|
||||
# have changed (window/unit/session) rather than on every pan.
|
||||
window = self._source.envelope(
|
||||
buckets=self._overview_points(), unit=self._unit
|
||||
)
|
||||
values = slice_.value_for_unit(self._unit)
|
||||
values = np.asarray(window.values, dtype=float)
|
||||
column = values[:, 0]
|
||||
finite = np.isfinite(column)
|
||||
self.overview_curve.setData(slice_.times[finite], column[finite])
|
||||
self.overview_curve.setData(window.times[finite], column[finite])
|
||||
self.overview.setXRange(first, last, padding=0.0)
|
||||
if self._mode == "browse":
|
||||
self.overview_region.setRegion(self._window)
|
||||
@@ -616,20 +638,25 @@ class ForcePlotsPanel(QFrame):
|
||||
self.btn_overview.setEnabled(self._store is not None)
|
||||
|
||||
def _reload_from_store(self, store) -> None:
|
||||
"""Re-populate the visible buffer from the whole-session store."""
|
||||
"""Re-populate the visible buffer from the session source."""
|
||||
window = self.history.window_sec
|
||||
slice_ = store.tail(window) if window > 0 else store.data()
|
||||
if len(slice_) == 0:
|
||||
first, last = self._source.extent()
|
||||
span = window if window > 0 else max(0.0, last - first)
|
||||
data = self._source.window(
|
||||
max(first, last - span), last, unit=self._unit
|
||||
)
|
||||
if len(data) == 0:
|
||||
return
|
||||
values = slice_.value_for_unit(self._unit)
|
||||
for index in range(len(slice_)):
|
||||
if not bool(slice_.valid[index]):
|
||||
self._availability_from_window(data)
|
||||
values = np.asarray(data.values, dtype=float)
|
||||
for index in range(len(data)):
|
||||
if not bool(data.valid[index]):
|
||||
continue
|
||||
row = values[index]
|
||||
if not all(np.isfinite(v) for v in row):
|
||||
continue
|
||||
self.history.push(
|
||||
float(slice_.times[index]),
|
||||
float(data.times[index]),
|
||||
float(row[0]),
|
||||
float(row[1]),
|
||||
float(row[2]),
|
||||
@@ -641,6 +668,19 @@ class ForcePlotsPanel(QFrame):
|
||||
curve.setData(state.times, state.values[:, i])
|
||||
self.plots[i].setXRange(*state.x_range, padding=0.0)
|
||||
|
||||
def _availability_from_window(self, window) -> None:
|
||||
"""Mark which axes have a value in the active unit, from a read window.
|
||||
|
||||
Used after a unit switch: the value cards must not keep showing the
|
||||
previous unit's numbers, and an axis with no curve in this unit has to be
|
||||
a dash until a fresh sample arrives.
|
||||
"""
|
||||
if window is None or len(window) == 0:
|
||||
return
|
||||
values = np.asarray(window.values, dtype=float)
|
||||
finite = np.isfinite(values) & np.asarray(window.valid, dtype=bool)[:, None]
|
||||
self._available = tuple(bool(finite[:, index].any()) for index in range(3))
|
||||
|
||||
def _uses_fixed_y(self) -> bool:
|
||||
"""Fixed Y whenever we have limits for the active unit.
|
||||
|
||||
@@ -690,10 +730,21 @@ class ForcePlotsPanel(QFrame):
|
||||
|
||||
def _render_value_cards(self) -> None:
|
||||
fmt = "{value:.3f}" if self._unit == "N" else "{value:.1f}"
|
||||
for value_label, unit_label, value in zip(
|
||||
self.value_labels, self.value_units, self._last_values
|
||||
unavailable = t("chart.value_unavailable")
|
||||
for index, (value_label, unit_label, value) in enumerate(
|
||||
zip(self.value_labels, self.value_units, self._last_values)
|
||||
):
|
||||
if not self._available[index]:
|
||||
# An axis with no curve in this unit: a dash, never a raw count
|
||||
# presented as a newton value.
|
||||
value_label.setText(unavailable)
|
||||
unit_label.setText("")
|
||||
value_label.setToolTip(
|
||||
t("chart.value_unavailable_tip", channel=CHANNEL_LABELS[index])
|
||||
)
|
||||
continue
|
||||
value_label.setText(fmt.format(value=value))
|
||||
value_label.setToolTip("")
|
||||
unit_label.setText(unit_display(self._unit))
|
||||
self.unit_badge.setText(unit_display(self._unit))
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ video frame, the depth pane and the charts' cursor line together.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
@@ -45,6 +46,10 @@ except ImportError as exc: # pragma: no cover
|
||||
|
||||
CHANNEL_TOKENS = ("plot_fn", "plot_fx", "plot_fy")
|
||||
SLIDER_STEPS = 2000
|
||||
#: Upper bound on the points handed to the replay curves.
|
||||
CHART_MAX_POINTS = 4000
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PlaybackDialog(QDialog):
|
||||
@@ -75,6 +80,13 @@ class PlaybackDialog(QDialog):
|
||||
self._depth_pending_seq: int | None = None
|
||||
self._depth_pending_position: float | None = None
|
||||
self._depth_result_seq: int | None = None
|
||||
#: Which recording each answer belongs to (incremented on every open).
|
||||
self._record_serial = 0
|
||||
self._depth_pending_record: int | None = None
|
||||
self._depth_result_record: int | None = None
|
||||
self._depth_result_frame = -1
|
||||
#: Processors whose owning request was still running when it was retired.
|
||||
self._retired_recomputers: list = []
|
||||
self._depth_poll = QTimer(self)
|
||||
self._depth_poll.setInterval(120)
|
||||
self._depth_poll.timeout.connect(self._poll_depth)
|
||||
@@ -166,9 +178,16 @@ class PlaybackDialog(QDialog):
|
||||
self.warning_label.setText(t("playback.open_failed", error=str(exc)))
|
||||
|
||||
def open_recording(self, directory: Path) -> None:
|
||||
if self._recomputer is not None:
|
||||
self._recomputer.close()
|
||||
self._recomputer = None
|
||||
"""Open a recording. Any depth answer still in flight is invalidated."""
|
||||
# A running request belongs to the *previous* recording: its result must
|
||||
# never be pasted into the new one, and its processor must be closed by
|
||||
# the thread that is using it, not here.
|
||||
self._retire_recomputer()
|
||||
self._record_serial += 1
|
||||
self._depth_request_seq += 1
|
||||
self._depth_pending_seq = None
|
||||
self._depth_pending_position = None
|
||||
self._depth_error = None
|
||||
self._reader = RecordingReader(directory)
|
||||
self._recomputer = DepthRecomputer(self._reader)
|
||||
info = self._reader.info()
|
||||
@@ -188,19 +207,48 @@ class PlaybackDialog(QDialog):
|
||||
self._depth_result_seq = None
|
||||
self.set_position(0.0)
|
||||
|
||||
def _retire_recomputer(self) -> None:
|
||||
"""Hand the current processor to its worker, or close it here if idle.
|
||||
|
||||
Closing a processor that a worker thread is still reading is a use-after-
|
||||
free; only the thread that owns it may close it, so a busy one is marked
|
||||
for the worker to close when it finishes.
|
||||
"""
|
||||
recomputer, self._recomputer = self._recomputer, None
|
||||
if recomputer is None:
|
||||
return
|
||||
if self._depth_busy():
|
||||
self._retired_recomputers.append(recomputer)
|
||||
else:
|
||||
try:
|
||||
recomputer.close()
|
||||
except Exception: # noqa: BLE001 - closing is best effort
|
||||
logger.exception("Could not close the depth recomputer")
|
||||
|
||||
def _refresh_warnings(self) -> None:
|
||||
if self._reader is None:
|
||||
return
|
||||
info = self._reader.info()
|
||||
_ok, sdk_version = DepthRecomputer.sdk_available()
|
||||
# The reader compares the recorded config hash against the config file as
|
||||
# it is now, which is what the depth recomputation would actually use.
|
||||
codes = self._reader.replay_warnings(sdk_version=sdk_version)
|
||||
if not codes:
|
||||
self.warning_label.setText("")
|
||||
return
|
||||
self.warning_label.setText(
|
||||
t("playback.warnings", detail=", ".join(t(f"playback.warn_{c}") for c in codes))
|
||||
text = ""
|
||||
if codes:
|
||||
text = t(
|
||||
"playback.warnings",
|
||||
detail=", ".join(t(f"playback.warn_{c}") for c in codes),
|
||||
)
|
||||
if info.decimated:
|
||||
# The curves are a view, not every sample: say so rather than let the
|
||||
# operator count points.
|
||||
note = t(
|
||||
"playback.decimated",
|
||||
shown=len(info.times),
|
||||
total=info.total_rows,
|
||||
)
|
||||
text = f"{text} {note}" if text else note
|
||||
self.warning_label.setText(text)
|
||||
|
||||
# ------------------------------------------------------------------ drawing
|
||||
def _render_curves(self) -> None:
|
||||
@@ -211,11 +259,21 @@ class PlaybackDialog(QDialog):
|
||||
for curve in self.curves:
|
||||
curve.setData([], [])
|
||||
return
|
||||
# Same rule as the live charts: invalid or non-finite points are not
|
||||
# drawn, whichever view is asking.
|
||||
times, drawn, _mask = info.plottable(self._unit)
|
||||
# Same rule as the live charts, per axis: a point is drawn only where the
|
||||
# sample is valid and *that* axis is finite, so a single-axis calibration
|
||||
# does not hide the axis that is calibrated. For a long recording the
|
||||
# whole-session view is decimated while it is read from disk, in the unit
|
||||
# being shown.
|
||||
window_all = getattr(self._reader, "window_all", None)
|
||||
if callable(window_all):
|
||||
window = window_all(unit=self._unit, max_points=CHART_MAX_POINTS)
|
||||
for index, curve in enumerate(self.curves):
|
||||
curve.setData(times, drawn[:, index])
|
||||
times, values = window.plottable_axis(self._unit, index)
|
||||
curve.setData(times, values)
|
||||
return
|
||||
for index, curve in enumerate(self.curves):
|
||||
times, values = info.plottable_axis(self._unit, index)
|
||||
curve.setData(times, values)
|
||||
self.plot_widget.ci.layout.setRowStretchFactor(index, 1)
|
||||
self.plot_widget.getItem(index, 0).setXRange(
|
||||
float(info.times[0]), float(info.times[-1]), padding=0.02
|
||||
@@ -307,22 +365,40 @@ class PlaybackDialog(QDialog):
|
||||
self._depth_error = None
|
||||
self.depth_label.setText(t("playback.depth_running"))
|
||||
position = self._position
|
||||
# Bind the answer to this request: a result that arrives after the cursor
|
||||
# moved is discarded rather than pasted onto the new position.
|
||||
# Bind the answer to this request: the recording, the frame and a request
|
||||
# sequence. A result that arrives after the cursor moved — or after
|
||||
# another recording was opened — is discarded rather than pasted on.
|
||||
recomputer = self._recomputer
|
||||
record = self._record_serial
|
||||
frame = self._current_frame()
|
||||
self._depth_request_seq += 1
|
||||
seq = self._depth_request_seq
|
||||
self._depth_pending_seq = seq
|
||||
self._depth_pending_position = position
|
||||
self._depth_pending_record = record
|
||||
|
||||
def work() -> None:
|
||||
# Runs off the UI thread and touches no Qt object.
|
||||
# Runs off the UI thread and touches no Qt object. It captures the
|
||||
# processor it was given: the dialog may already be showing another
|
||||
# recording by the time this returns.
|
||||
try:
|
||||
result = self._recomputer.depth_at(position)
|
||||
result = recomputer.depth_at(position)
|
||||
except BaseException as exc: # noqa: BLE001 - reported via the poll
|
||||
self._depth_error = exc
|
||||
self._depth_error = (record, exc)
|
||||
return
|
||||
finally:
|
||||
if recomputer in self._retired_recomputers:
|
||||
# Retired while this request was running: the owning thread
|
||||
# closes it, not the UI thread.
|
||||
self._retired_recomputers.remove(recomputer)
|
||||
try:
|
||||
recomputer.close()
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("Could not close a retired recomputer")
|
||||
self._depth_result = result
|
||||
self._depth_result_seq = seq
|
||||
self._depth_result_record = record
|
||||
self._depth_result_frame = getattr(frame, "frame_id", -1)
|
||||
|
||||
self._depth_thread = threading.Thread(
|
||||
target=work, name="depth-replay", daemon=True
|
||||
@@ -337,15 +413,22 @@ class PlaybackDialog(QDialog):
|
||||
return
|
||||
self._depth_poll.stop()
|
||||
pending = self._depth_pending_seq
|
||||
record = self._depth_pending_record
|
||||
self._depth_pending_seq = None
|
||||
self._depth_pending_position = None
|
||||
self._depth_pending_record = None
|
||||
if self._depth_error is not None:
|
||||
error_record, exc = self._depth_error
|
||||
self._depth_error = None
|
||||
if error_record != self._record_serial:
|
||||
# The failure belongs to a recording that is no longer open.
|
||||
return
|
||||
self._depth_ok = False
|
||||
self.depth_label.setPixmap(QPixmap())
|
||||
self.depth_label.setText(
|
||||
t("playback.depth_failed", detail=str(self._depth_error))
|
||||
)
|
||||
self._depth_error = None
|
||||
self.depth_label.setText(t("playback.depth_failed", detail=str(exc)))
|
||||
return
|
||||
if record != self._record_serial:
|
||||
# Another recording was opened while this was running.
|
||||
return
|
||||
result = self._depth_result
|
||||
if result is None or self._depth_result_seq != pending:
|
||||
@@ -468,10 +551,18 @@ class PlaybackDialog(QDialog):
|
||||
def closeEvent(self, event) -> None: # noqa: N802
|
||||
self._depth_poll.stop()
|
||||
# A depth recomputation that is blocked inside the SDK cannot be
|
||||
# cancelled; it is a daemon thread, so it does not keep the process
|
||||
# alive and the window always closes.
|
||||
if self._recomputer is not None and not self._depth_busy():
|
||||
self._recomputer.close()
|
||||
# cancelled; it is a daemon thread, so it does not keep the process alive
|
||||
# and the window always closes. The processor is closed by whichever
|
||||
# thread owns it (see ``_retire_recomputer``), never by the UI thread
|
||||
# while a worker is reading it.
|
||||
for recomputer in list(self._retired_recomputers):
|
||||
if not self._depth_busy():
|
||||
self._retired_recomputers.remove(recomputer)
|
||||
try:
|
||||
recomputer.close()
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("Could not close a retired recomputer")
|
||||
self._retire_recomputer()
|
||||
super().closeEvent(event)
|
||||
|
||||
def _depth_busy(self) -> bool:
|
||||
|
||||
@@ -289,7 +289,7 @@ def test_cal_panel_records_and_lists_samples(make_window):
|
||||
]
|
||||
|
||||
|
||||
def _add_calibration_point(window, *, load: float, raw: float, samples: int = 40) -> None:
|
||||
def _add_calibration_point(window, *, load: float, raw: float, samples: int = 40):
|
||||
"""Drive one point through the recorder and into the panel's table."""
|
||||
from orisys_app_shared.force.session import SessionSample
|
||||
|
||||
@@ -305,7 +305,9 @@ def _add_calibration_point(window, *, load: float, raw: float, samples: int = 40
|
||||
valid=True,
|
||||
)
|
||||
)
|
||||
window.cal_panel.on_record_finished(window.cal_recorder.finish_now())
|
||||
point = window.cal_recorder.finish_now()
|
||||
window.cal_panel.on_record_finished(point)
|
||||
return point
|
||||
|
||||
|
||||
def test_calibration_diagnostics_drive_the_fit(make_window):
|
||||
@@ -357,6 +359,64 @@ def test_calibration_diagnostics_drive_the_fit(make_window):
|
||||
model = build_export_model({key: store.channel(key) for key in ("fn", "fx", "fy")})
|
||||
assert abs(predict_channel(model, "fn", 400.0) - 2.0) > 0.05
|
||||
|
||||
# The store, the label and the export are the same preparation: the panel
|
||||
# must not derive the model twice and show numbers for one of them.
|
||||
prepared = window.cal_panel._prepared
|
||||
assert prepared is not None and prepared.ok
|
||||
assert list(store.channel("fn")) == list(prepared.samples["fn"])
|
||||
assert prepared.points_in_fit("fn") == 3
|
||||
|
||||
|
||||
def test_the_panel_shows_the_numbers_of_the_file_it_exports(make_window, record_root):
|
||||
"""Displayed diagnostics must match the written coefficients.
|
||||
|
||||
A repeated reference load makes this sharp: the two samples of one load must
|
||||
be merged before the build, so the exported curve and the panel's residual
|
||||
and validation figures all come from that single prepared model.
|
||||
"""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import orisys.calibration as sdk_calibration
|
||||
|
||||
from orisys_app_shared.force.calibration_io import build_force_calibration_v2
|
||||
|
||||
window = _show(make_window(), (1400, 880))
|
||||
window.cal_panel.set_running(True)
|
||||
window.cal_panel.set_meta_context(sensor_type="box", config_ref=None)
|
||||
_add_calibration_point(window, load=1.0, raw=100.0)
|
||||
_add_calibration_point(window, load=2.0, raw=200.0)
|
||||
_add_calibration_point(window, load=2.0, raw=240.0) # a deliberate repeat
|
||||
_add_calibration_point(window, load=4.0, raw=400.0)
|
||||
_add_calibration_point(window, load=3.0, raw=290.0)
|
||||
|
||||
panel = window.cal_panel
|
||||
panel.table.setCurrentCell(4, 0)
|
||||
panel._on_toggle_validation()
|
||||
panel._rebuild_store()
|
||||
panel._refresh_fit_label()
|
||||
prepared = panel._prepared
|
||||
assert prepared is not None and prepared.ok
|
||||
assert len(prepared.samples["fn"]) == 3, "the repeated load was not merged"
|
||||
|
||||
# Write the document the export would write, load it back with the SDK and
|
||||
# check that its predictions are the prepared ones.
|
||||
document, _warnings = build_force_calibration_v2(prepared)
|
||||
path = Path(record_root) / "force_calibration.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
loaded = sdk_calibration.load(path, strict=True)
|
||||
|
||||
for point in window.cal_recorder.points:
|
||||
raw = float(point.loaded_stats().mean)
|
||||
predicted = float(loaded.get("fn").convert(raw))
|
||||
assert predicted == pytest.approx(prepared.predict("fn", raw), abs=1e-9)
|
||||
|
||||
# ...and the label describes that same model.
|
||||
label = panel.fit_label.text()
|
||||
assert "分段线性" in label or "piecewise" in label, label
|
||||
assert "3" in label
|
||||
assert prepared.max_validation_error("fn") > 0.0
|
||||
|
||||
|
||||
def test_calibration_point_dialog_shows_three_axes(make_window):
|
||||
window = _show(make_window(), (1400, 880))
|
||||
@@ -373,6 +433,137 @@ def test_calibration_point_dialog_shows_three_axes(make_window):
|
||||
dialog.close()
|
||||
|
||||
|
||||
def _write_session_file(root, *, sensor_type, sn, config_ref, points):
|
||||
import json
|
||||
|
||||
from orisys_app_shared.force.calibration_session import build_session_document
|
||||
|
||||
document = build_session_document(
|
||||
points,
|
||||
meta={"sensor_type": sensor_type, "sn": sn, "config_ref": config_ref},
|
||||
)
|
||||
path = Path(root) / f"calibration_session_{sensor_type}.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def test_opening_device_a_record_keeps_device_a_identity(make_window, record_root):
|
||||
"""Device A's record must not be restamped with device B's identity."""
|
||||
import json
|
||||
from unittest.mock import patch as _patch
|
||||
|
||||
import orisys_studio.ui.cal_panel as cal_panel_module
|
||||
from orisys_app_shared.force.calibration_io import build_force_calibration_v2
|
||||
|
||||
window = _show(make_window(), (1400, 880))
|
||||
panel = window.cal_panel
|
||||
# A record taken on device A, with a full audit series.
|
||||
session_points = [
|
||||
_add_calibration_point(window, load=1.0, raw=100.0),
|
||||
_add_calibration_point(window, load=3.0, raw=300.0),
|
||||
]
|
||||
source = _write_session_file(
|
||||
record_root,
|
||||
sensor_type="box",
|
||||
sn="SN-A",
|
||||
config_ref="D:/lab/deviceA/sensor.json",
|
||||
points=session_points,
|
||||
)
|
||||
window.cal_recorder.clear_points()
|
||||
|
||||
# ...and the live context is device B.
|
||||
panel.set_meta_context(sensor_type="finger", config_ref="D:/lab/deviceB/sensor.json")
|
||||
panel.sn_edit.setText("SN-B")
|
||||
|
||||
panel._on_open_record(str(source))
|
||||
assert panel.sn_edit.text() == "SN-A", "the record's SN was overwritten"
|
||||
meta = panel._export_meta()
|
||||
assert meta["sensor_type"] == "box"
|
||||
assert meta["sn"] == "SN-A"
|
||||
assert meta["config_ref"] == "D:/lab/deviceA/sensor.json"
|
||||
assert meta["source_record"].endswith(source.name)
|
||||
|
||||
# Saving the record again still says A.
|
||||
import orisys_studio.calibration_service as cal_service_module
|
||||
|
||||
with _patch.object(cal_panel_module, "calibration_export_dir", lambda: Path(record_root)):
|
||||
with _patch.object(
|
||||
cal_service_module, "calibration_export_dir", lambda: Path(record_root)
|
||||
):
|
||||
with _patch.object(cal_panel_module.QMessageBox, "information", lambda *a, **k: None):
|
||||
panel._on_save_record()
|
||||
saved = sorted(Path(record_root).glob("calibration_session_box_*.json"))
|
||||
assert saved, "the record was not saved under device A's type"
|
||||
saved_meta = json.loads(saved[-1].read_text(encoding="utf-8"))["meta"]
|
||||
assert saved_meta["sn"] == "SN-A"
|
||||
assert saved_meta["sensor_type"] == "box"
|
||||
assert saved_meta["config_ref"] == "D:/lab/deviceA/sensor.json"
|
||||
|
||||
# ...and so does the exported coefficient document.
|
||||
document, _warnings = build_force_calibration_v2(panel._prepared)
|
||||
assert document["meta"]["sensor_type"] == "box"
|
||||
assert document["meta"]["sn"] == "SN-A"
|
||||
assert document["meta"]["config_ref"] == "D:/lab/deviceA/sensor.json"
|
||||
|
||||
|
||||
def test_a_reopened_point_can_be_inspected_from_its_samples(make_window, record_root):
|
||||
"""A record with samples but no cached trace still opens its diagnostics."""
|
||||
window = _show(make_window(), (1400, 880))
|
||||
panel = window.cal_panel
|
||||
points = [
|
||||
_add_calibration_point(window, load=1.0, raw=100.0),
|
||||
_add_calibration_point(window, load=2.0, raw=200.0),
|
||||
]
|
||||
import json
|
||||
|
||||
from orisys_app_shared.force.calibration_session import build_session_document
|
||||
|
||||
document = build_session_document(points, meta={"sensor_type": "box"})
|
||||
for entry in document["points"]:
|
||||
entry["trace"] = {} # only the audit series survived
|
||||
entry["trace_dt_s"] = 0.0
|
||||
source = Path(record_root) / "no_trace.json"
|
||||
source.write_text(json.dumps(document), encoding="utf-8")
|
||||
|
||||
panel._on_open_record(str(source))
|
||||
reopened = window.cal_recorder.points[0]
|
||||
assert reopened.samples, "the reopened point lost its samples"
|
||||
dialog = panel.point_dialog(reopened)
|
||||
assert dialog is not None, "the reopened point could not be inspected"
|
||||
assert len(dialog.plot_widget.ci.items) == 3
|
||||
dialog.close()
|
||||
|
||||
|
||||
def test_applying_another_devices_calibration_is_flagged(make_window, record_root):
|
||||
"""The apply path states a sensor/config mismatch instead of silently applying."""
|
||||
import json
|
||||
|
||||
from orisys_app_shared.force.calibration_io import build_export_model, write_force_calibration
|
||||
from orisys_app_shared.force.calibration_io import CalSample, CalSessionStore
|
||||
|
||||
window, stuck = _running_window(make_window)
|
||||
store = CalSessionStore()
|
||||
store.aggregate("fn", CalSample(weight_n=0.0, raw_mean=0.0, n=10))
|
||||
store.aggregate("fn", CalSample(weight_n=5.0, raw_mean=500.0, n=10))
|
||||
model = build_export_model({key: store.channel(key) for key in ("fn", "fx", "fy")})
|
||||
document = model.to_dict()
|
||||
document["identity"] = {"sensor_type": "box", "sn": "SN-A"}
|
||||
document["pipeline"] = {"config_ref": "D:/lab/deviceA/sensor.json"}
|
||||
path = Path(record_root) / "deviceA_calibration.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
|
||||
# The window is connected to a finger device.
|
||||
mismatch = window.calibration_target_mismatch(
|
||||
str(path), sensor_type="finger", config_ref="D:/lab/deviceB/sensor.json"
|
||||
)
|
||||
assert any("sensor_type" in reason for reason in mismatch), mismatch
|
||||
assert any("config_ref" in reason for reason in mismatch), mismatch
|
||||
assert not window.calibration_target_mismatch(
|
||||
str(path), sensor_type="box", config_ref="D:/lab/deviceA/sensor.json"
|
||||
)
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_force_plots_render_demo_data(make_window):
|
||||
window = _show(make_window(), (1440, 880))
|
||||
for i in range(60):
|
||||
@@ -502,6 +693,344 @@ def test_history_keeps_negative_fn(make_window):
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_the_live_axis_is_session_relative_and_survives_a_unit_switch(make_window):
|
||||
"""The live buffer, the store and a rebuilt history share one time base.
|
||||
|
||||
Feeding the wall clock into the plot put the live axis ~1.7e9 s away from the
|
||||
session store's, so a unit switch (which rebuilds from the store) jumped the
|
||||
X axis and the chart looked like it had lost its history.
|
||||
"""
|
||||
wall = 1_700_000_000.0
|
||||
window, stuck = _running_window(make_window)
|
||||
window.toolbar.units_combo.setCurrentIndex(0) # newton
|
||||
window.force_plots.reset()
|
||||
for index in range(90):
|
||||
stuck.emit(
|
||||
ui_check._fake_packet(
|
||||
index, timestamp=wall + index / 30.0, mono_t=index / 30.0
|
||||
)
|
||||
)
|
||||
window._on_run_tick()
|
||||
|
||||
before = window.force_plots.history.state()
|
||||
assert before is not None
|
||||
end_before = float(before.times[-1])
|
||||
assert end_before < 60.0, (
|
||||
f"the live X axis is not session-relative: it ends at {end_before}"
|
||||
)
|
||||
assert abs(end_before - 89 / 30.0) < 0.5, end_before
|
||||
|
||||
for units_index in (1, 0, 1, 0): # raw -> N -> raw -> N
|
||||
window.toolbar.units_combo.setCurrentIndex(units_index)
|
||||
|
||||
after = window.force_plots.history.state()
|
||||
assert after is not None, "the unit switch emptied the rebuilt history"
|
||||
assert abs(float(after.times[-1]) - end_before) < 1.0, (
|
||||
f"the X axis jumped across a unit switch: {end_before} -> {after.times[-1]}"
|
||||
)
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_a_single_calibrated_axis_is_shown_and_the_others_are_unavailable(make_window):
|
||||
"""Only FN calibrated: FN in newtons, FX/FY explicitly unavailable."""
|
||||
import math
|
||||
|
||||
from orisys_app_shared.i18n import t
|
||||
|
||||
window, stuck = _running_window(make_window)
|
||||
window.toolbar.units_combo.setCurrentIndex(0) # newton
|
||||
window.force_plots.reset()
|
||||
for index in range(10):
|
||||
stuck.emit(
|
||||
ui_check._fake_packet(
|
||||
index,
|
||||
calibrated_axes=("fn",),
|
||||
shear_n=float("nan"),
|
||||
)
|
||||
)
|
||||
window._on_run_tick()
|
||||
|
||||
plots = window.force_plots
|
||||
assert plots._available == (True, False, False), plots._available
|
||||
assert plots.value_labels[0].text() != t("chart.value_unavailable")
|
||||
for index in (1, 2):
|
||||
assert plots.value_labels[index].text() == t("chart.value_unavailable"), (
|
||||
"an axis with no curve must not show a number"
|
||||
)
|
||||
state = plots.history.state()
|
||||
assert math.isfinite(float(state.values[:, 0].max()))
|
||||
assert not any(math.isfinite(v) for v in state.values[:, 1]), (
|
||||
"FX carried a value in newton mode without a curve"
|
||||
)
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_a_slow_close_is_not_reported_as_a_finished_recording(make_window):
|
||||
"""Stopping must not present an unclosed file as a finished experiment."""
|
||||
import threading as _threading
|
||||
from unittest.mock import patch as _patch
|
||||
|
||||
from orisys_app_shared.i18n import t
|
||||
from orisys_app_shared.record.experiment import _SessionWriter
|
||||
|
||||
window, stuck = _running_window(make_window)
|
||||
window.toolbar.btn_record.setChecked(True)
|
||||
stuck.emit(ui_check._fake_packet(0))
|
||||
window._on_run_tick()
|
||||
|
||||
started = _threading.Event()
|
||||
release = _threading.Event()
|
||||
original = _SessionWriter.write_sample
|
||||
|
||||
def slow(self, *args):
|
||||
started.set()
|
||||
release.wait(10.0)
|
||||
return original(self, *args)
|
||||
|
||||
with _patch.object(_SessionWriter, "write_sample", slow):
|
||||
stuck.emit(ui_check._fake_packet(1))
|
||||
window._on_run_tick()
|
||||
assert started.wait(5.0), "the writer never picked up the frame"
|
||||
|
||||
window._stop_recording(wait=False)
|
||||
assert window.recorder.report is None, "the stop claimed a finished recording"
|
||||
assert window._record_report_pending, "the pending report was not tracked"
|
||||
assert window._record_last_report is None
|
||||
assert not window.toolbar.btn_record.isChecked(), (
|
||||
"the record button still looks active while the file closes"
|
||||
)
|
||||
assert window.statusBar().currentMessage() == t("record.closing")
|
||||
|
||||
release.set()
|
||||
deadline = __import__("time").monotonic() + 10.0
|
||||
while window.recorder.report is None and __import__("time").monotonic() < deadline:
|
||||
__import__("time").sleep(0.01)
|
||||
window._on_run_tick()
|
||||
|
||||
report = window._record_last_report
|
||||
assert report is not None and report.completed, "the terminal report was never published"
|
||||
assert not window._record_report_pending
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_a_failed_recording_is_consumed_even_when_it_is_not_running(make_window):
|
||||
"""A recording that died on its own still has a report to publish."""
|
||||
import time as _time
|
||||
from unittest.mock import patch as _patch
|
||||
|
||||
from orisys_app_shared.record.experiment import _SessionWriter
|
||||
|
||||
window, stuck = _running_window(make_window)
|
||||
window.toolbar.btn_record.setChecked(True)
|
||||
assert window.recorder.is_recording
|
||||
|
||||
with _patch.object(_SessionWriter, "write_sample", side_effect=OSError("disk full")):
|
||||
for index in range(8):
|
||||
stuck.emit(ui_check._fake_packet(index))
|
||||
window._on_run_tick()
|
||||
deadline = _time.monotonic() + 3.0
|
||||
while window.recorder.stats().write_errors < 5 and _time.monotonic() < deadline:
|
||||
_time.sleep(0.01)
|
||||
assert window.recorder.stats().failed
|
||||
assert not window.recorder.is_recording
|
||||
|
||||
# ``is_recording`` is already False: the stop must still consume the report.
|
||||
window._stop_recording()
|
||||
report = window._record_last_report
|
||||
assert report is not None and report.stats.failed
|
||||
assert not window.toolbar.btn_record.isChecked()
|
||||
assert window._session_phase == "error"
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_the_depth_request_belongs_to_the_recording_it_started_in(make_window, record_root):
|
||||
"""Opening another recording must invalidate the request still running."""
|
||||
import threading as _threading
|
||||
import time as _time
|
||||
|
||||
from PyQt5.QtWidgets import QApplication
|
||||
|
||||
from orisys_app_shared.record.depth import DepthResult
|
||||
from orisys_studio.ui.playback_dialog import PlaybackDialog
|
||||
|
||||
window = _show(make_window(), (1200, 700))
|
||||
dialog = PlaybackDialog(theme="dark", directory=Path(record_root))
|
||||
first_serial = dialog._record_serial
|
||||
|
||||
started = _threading.Event()
|
||||
release = _threading.Event()
|
||||
closed: list[str] = []
|
||||
|
||||
class _Blocking:
|
||||
def preflight(self) -> str:
|
||||
return ""
|
||||
|
||||
def depth_at(self, _position, **_kwargs):
|
||||
started.set()
|
||||
release.wait(10.0)
|
||||
return DepthResult(
|
||||
depth=np.ones((2, 2)), frame_id=0, t_rel_s=0.0, record_id="old"
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
closed.append("closed")
|
||||
|
||||
blocking = _Blocking()
|
||||
dialog._recomputer = blocking
|
||||
dialog._on_recompute_depth()
|
||||
assert started.wait(5.0), "the depth request never started"
|
||||
|
||||
# Another recording is opened while the request is in flight.
|
||||
dialog.open_recording(Path(record_root))
|
||||
assert dialog._record_serial == first_serial + 1
|
||||
assert dialog._recomputer is not blocking, "the processor was not replaced"
|
||||
assert closed == [], "the UI thread closed a processor a worker was reading"
|
||||
|
||||
release.set()
|
||||
deadline = _time.monotonic() + 10.0
|
||||
while not closed and _time.monotonic() < deadline:
|
||||
QApplication.processEvents()
|
||||
_time.sleep(0.01)
|
||||
assert closed == ["closed"], "the retired processor was never closed"
|
||||
assert dialog._depth_thread is not None and not dialog._depth_thread.is_alive()
|
||||
|
||||
# The answer belongs to the previous recording, so it is dropped, not drawn.
|
||||
dialog._poll_depth()
|
||||
QApplication.processEvents()
|
||||
assert dialog._depth_ok is False
|
||||
pixmap = dialog.depth_label.pixmap()
|
||||
assert pixmap is None or pixmap.isNull(), "an answer for another recording was drawn"
|
||||
dialog.close()
|
||||
window.close()
|
||||
|
||||
|
||||
def test_a_busy_depth_processor_is_not_closed_by_the_ui_thread(make_window, record_root):
|
||||
import threading as _threading
|
||||
import time as _time
|
||||
|
||||
from PyQt5.QtWidgets import QApplication
|
||||
|
||||
from orisys_app_shared.record.depth import DepthResult
|
||||
from orisys_studio.ui.playback_dialog import PlaybackDialog
|
||||
|
||||
window = _show(make_window(), (1200, 700))
|
||||
dialog = PlaybackDialog(theme="dark", directory=Path(record_root))
|
||||
started = _threading.Event()
|
||||
release = _threading.Event()
|
||||
closed: list[str] = []
|
||||
|
||||
class _Blocking:
|
||||
def preflight(self) -> str:
|
||||
return ""
|
||||
|
||||
def depth_at(self, _position, **_kwargs):
|
||||
started.set()
|
||||
release.wait(10.0)
|
||||
return DepthResult(depth=np.ones((2, 2)), frame_id=0, t_rel_s=0.0)
|
||||
|
||||
def close(self) -> None:
|
||||
closed.append("closed")
|
||||
|
||||
dialog._recomputer = _Blocking()
|
||||
dialog._on_recompute_depth()
|
||||
assert started.wait(5.0)
|
||||
dialog.close() # closing the window must not free a processor in use
|
||||
QApplication.processEvents()
|
||||
assert closed == [], "the window closed a processor that was still reading"
|
||||
release.set()
|
||||
deadline = _time.monotonic() + 10.0
|
||||
while not closed and _time.monotonic() < deadline:
|
||||
QApplication.processEvents()
|
||||
_time.sleep(0.01)
|
||||
assert closed == ["closed"], "the owning thread never closed the processor"
|
||||
window.close()
|
||||
|
||||
|
||||
def test_a_confirmed_calibration_is_snapshotted_into_the_recording(make_window, record_root):
|
||||
"""The recording keeps the coefficients themselves, not only their hash."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import orisys_studio.app_window as app_module
|
||||
from orisys_app_shared.i18n import t
|
||||
|
||||
window, stuck = _running_window(make_window)
|
||||
app_module.recordings_dir = lambda: record_root
|
||||
window.toolbar.btn_record.setChecked(True)
|
||||
assert window.recorder.is_recording
|
||||
|
||||
coefficients = Path(record_root) / "force_calibration_box.json"
|
||||
coefficients.write_text('{"unit": "N", "channels": {}}', encoding="utf-8")
|
||||
version = "deadbeefcafe1234"
|
||||
window._on_calibration_applied(
|
||||
str(coefficients), version, 21, 0.7, int(window.session_store.session_id)
|
||||
)
|
||||
window._drain_calibration_events()
|
||||
|
||||
directory = window.recorder.directory
|
||||
assert directory is not None
|
||||
snapshots = sorted(path.name for path in (directory / "calibration").iterdir())
|
||||
assert snapshots == [f"force_calibration_{version[:12]}.json"], snapshots
|
||||
assert (directory / "calibration" / snapshots[0]).read_text(
|
||||
encoding="utf-8"
|
||||
) == '{"unit": "N", "channels": {}}'
|
||||
assert window._active_calibration_id == version
|
||||
assert window.session_store.calibration_id == version
|
||||
|
||||
window._stop_recording()
|
||||
meta = json.loads((directory / "session.json").read_text(encoding="utf-8"))
|
||||
entries = meta["calibration_snapshots"]
|
||||
assert entries and entries[0]["version"] == version
|
||||
assert entries[0]["file"] == f"calibration/{snapshots[0]}"
|
||||
assert entries[0]["frame_id"] == 21
|
||||
event = next(e for e in meta["events"] if e["kind"] == "calibration_applied")
|
||||
assert event["frame_id"] == 21 and event["mono_t"] == pytest.approx(0.7)
|
||||
assert snapshots[0] in event["detail"]
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_a_confirmed_absence_of_calibration_does_not_keep_the_old_version(make_window):
|
||||
"""An empty version is the honest answer; it must not fall back to the path."""
|
||||
from orisys_app_shared.i18n import t
|
||||
|
||||
window = _show(make_window(), (1400, 880))
|
||||
window._active_calibration_id = "previous-hash"
|
||||
window._on_calibration_applied("/tmp/whatever.json", "", 5, 0.2, 1)
|
||||
window._drain_calibration_events()
|
||||
assert window._active_calibration_id == "", (
|
||||
"an uncalibrated device kept claiming the previous coefficients"
|
||||
)
|
||||
assert window.statusBar().currentMessage() == t("cal.applied_none")
|
||||
|
||||
|
||||
def test_a_zeroing_is_recorded_only_after_the_worker_confirms_it(make_window, record_root):
|
||||
import json
|
||||
|
||||
import orisys_studio.app_window as app_module
|
||||
|
||||
window, stuck = _running_window(make_window)
|
||||
app_module.recordings_dir = lambda: record_root
|
||||
window.toolbar.btn_record.setChecked(True)
|
||||
stuck.emit(ui_check._fake_packet(3))
|
||||
window._on_run_tick()
|
||||
|
||||
window._on_reset()
|
||||
assert stuck.reset_requests == 1, "the zeroing was not requested"
|
||||
# The stub confirms from the acquisition side, as a real device does.
|
||||
stuck.on_reset_applied = window._on_reset_applied
|
||||
window._on_reset()
|
||||
window._on_run_tick()
|
||||
window._stop_recording()
|
||||
|
||||
directory = window.recorder.directory
|
||||
meta = json.loads((directory / "session.json").read_text(encoding="utf-8"))
|
||||
zeroing = [e for e in meta["events"] if e["kind"] == "zeroing"]
|
||||
assert len(zeroing) == 1, f"unexpected zeroing events: {zeroing}"
|
||||
assert zeroing[0]["frame_id"] == 3, zeroing[0]
|
||||
assert zeroing[0]["mono_t"] is not None
|
||||
_finish(window, stuck)
|
||||
|
||||
|
||||
def test_depth_view_reports_missing_data_and_fixes_its_scale(make_window):
|
||||
import numpy as np
|
||||
|
||||
|
||||
@@ -144,6 +144,10 @@ def _fake_packet(
|
||||
valid: bool = True,
|
||||
depth=None,
|
||||
reason: str = "",
|
||||
calibrated_axes=None,
|
||||
timestamp: float | None = None,
|
||||
mono_t: float | None = None,
|
||||
shear_n: float | None = None,
|
||||
):
|
||||
"""A packet shaped like the real worker's, for the record-channel checks."""
|
||||
import numpy as np
|
||||
@@ -157,17 +161,19 @@ def _fake_packet(
|
||||
return SdkPacket(
|
||||
raw_frame=blank,
|
||||
processed_frame=blank,
|
||||
timestamp=1000.0 + frame_id / 30.0,
|
||||
timestamp=1000.0 + frame_id / 30.0 if timestamp is None else timestamp,
|
||||
mono_t=mono_t,
|
||||
fps=30.0,
|
||||
process_ms=1.0,
|
||||
normal_force=(2.0 + frame_id * 0.01) if valid else nan,
|
||||
shear_x=0.5 if valid else nan,
|
||||
shear_x=(0.5 if shear_n is None else shear_n) if valid else nan,
|
||||
shear_y=-0.5 if valid else nan,
|
||||
normal_force_raw=(9.0 + frame_id * 0.1) if valid else nan,
|
||||
shear_x_raw=0.4 if valid else nan,
|
||||
shear_y_raw=-0.4 if valid else nan,
|
||||
force_unit="N",
|
||||
force_calibrated=True,
|
||||
calibrated_axes=tuple(calibrated_axes or ()),
|
||||
session_id=1,
|
||||
frame_id=frame_id,
|
||||
valid=valid,
|
||||
@@ -194,6 +200,27 @@ class _StuckWorker(RecordSinkMixin):
|
||||
self.stop_blocked_for = 0.0
|
||||
#: Released by ``release()`` so daemon threads do not outlive the run.
|
||||
self._release = threading.Event()
|
||||
#: Calibration requests this worker was asked to apply.
|
||||
self.calibration_requests: list[str] = []
|
||||
#: Zeroing requests, and the confirmation callback the window installs.
|
||||
self.reset_requests = 0
|
||||
self.on_reset_applied = None
|
||||
|
||||
def set_pending_calibration(self, path) -> None:
|
||||
self.calibration_requests.append(str(path))
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Request a zeroing; a real device confirms it from its own thread.
|
||||
|
||||
This stub confirms immediately (a fast device), so the window's event path
|
||||
— request, confirmation, recorded event — is exercised end to end.
|
||||
"""
|
||||
self.reset_requests += 1
|
||||
if callable(self.on_reset_applied):
|
||||
packet = self._latest
|
||||
frame_id = int(getattr(packet, "frame_id", -1))
|
||||
mono_t = float(getattr(packet, "mono_t", 0.0) or 0.0)
|
||||
self.on_reset_applied(frame_id, mono_t, 1)
|
||||
|
||||
def start(self, _source, *, config_ref=None) -> None:
|
||||
self._release.clear()
|
||||
@@ -229,9 +256,6 @@ class _StuckWorker(RecordSinkMixin):
|
||||
reason, self._end_reason = self._end_reason, None
|
||||
return reason
|
||||
|
||||
def reset(self) -> None:
|
||||
return None
|
||||
|
||||
def stop(self, *, wait: bool = False) -> None:
|
||||
self._end_reason = self._end_reason or "stopped"
|
||||
started = time.perf_counter()
|
||||
@@ -666,8 +690,10 @@ def main() -> int:
|
||||
)
|
||||
from orisys_app_shared.force.calibration_io import (
|
||||
build_export_model,
|
||||
build_force_calibration_v2 as _build_v2,
|
||||
predict_channel,
|
||||
)
|
||||
import orisys.calibration as _sdk_calibration
|
||||
|
||||
store = window.cal_panel._store
|
||||
model = build_export_model(
|
||||
@@ -678,6 +704,44 @@ def main() -> int:
|
||||
abs(predicted - 2.0) > 0.05,
|
||||
f"excluding a point left the exported model unchanged: {predicted}",
|
||||
)
|
||||
# One preparation: the store the panel shows is the prepared model's
|
||||
# input, and the export serialises that same object.
|
||||
prepared = window.cal_panel._prepared
|
||||
check(
|
||||
prepared is not None and prepared.ok,
|
||||
"the panel has no prepared calibration to show or export",
|
||||
)
|
||||
check(
|
||||
list(store.channel("fn")) == list(prepared.samples["fn"]),
|
||||
"the panel store is not the prepared model's input",
|
||||
)
|
||||
doc, _warnings = _build_v2(prepared)
|
||||
export_dir = APP_ROOT / "outputs" / "_cal_export_check"
|
||||
if export_dir.exists():
|
||||
for stale in export_dir.iterdir():
|
||||
stale.unlink()
|
||||
export_dir.mkdir(parents=True, exist_ok=True)
|
||||
written = export_dir / "exported_calibration.json"
|
||||
import json as _cal_json
|
||||
|
||||
written.write_text(_cal_json.dumps(doc), encoding="utf-8")
|
||||
loaded = _sdk_calibration.load(written, strict=True)
|
||||
for point in window.cal_recorder.points:
|
||||
if point.excluded:
|
||||
continue
|
||||
raw = float(point.loaded_stats().mean)
|
||||
check(
|
||||
abs(
|
||||
float(loaded.get("fn").convert(raw))
|
||||
- prepared.predict("fn", raw)
|
||||
)
|
||||
<= 1e-9,
|
||||
"the exported file and the panel disagree about the model",
|
||||
)
|
||||
check(
|
||||
len(prepared.samples["fn"]) == 3,
|
||||
f"the prepared build has {len(prepared.samples['fn'])} points, not 3",
|
||||
)
|
||||
window.cal_recorder.reset_session()
|
||||
window.cal_panel._refresh_table()
|
||||
|
||||
@@ -961,7 +1025,8 @@ def main() -> int:
|
||||
app.processEvents()
|
||||
check(window.recorder.is_recording, "the recording did not start")
|
||||
check(
|
||||
window.toolbar.btn_record.isEnabled() and window.toolbar.btn_record.text() == t("record.stop_label"),
|
||||
window.toolbar.btn_record.isEnabled()
|
||||
and window.toolbar.btn_record.text() == t("record.stop_label"),
|
||||
"the record button is not a usable stop control",
|
||||
)
|
||||
check(
|
||||
@@ -974,13 +1039,26 @@ def main() -> int:
|
||||
window._on_run_tick()
|
||||
stuck.emit(_fake_packet(90, valid=False, reason="read failed"))
|
||||
window._on_run_tick()
|
||||
window._on_reset() # event: zeroing
|
||||
# The zeroing is a *request*; the event is recorded only once the acquisition
|
||||
# thread confirms it took effect (the stub confirms immediately).
|
||||
stuck.on_reset_applied = window._on_reset_applied
|
||||
window._on_reset()
|
||||
window._on_run_tick()
|
||||
app.processEvents()
|
||||
|
||||
window.toolbar.btn_record.setChecked(False)
|
||||
app.processEvents()
|
||||
report = window._record_last_report
|
||||
check(not window.recorder.is_recording, "the recording did not stop")
|
||||
check(report is not None and report.ok, f"recording reported problems: {getattr(report, 'problems', None)}")
|
||||
check(
|
||||
not window.toolbar.btn_record.isChecked(),
|
||||
"the record button still looks active after the recording stopped",
|
||||
)
|
||||
check(
|
||||
window._record_report_pending is False,
|
||||
"a finished recording is still waiting for a report",
|
||||
)
|
||||
|
||||
import csv as _csv
|
||||
import json as _json
|
||||
@@ -1019,6 +1097,388 @@ def main() -> int:
|
||||
stuck.release()
|
||||
_shutil.rmtree(scratch, ignore_errors=True)
|
||||
|
||||
section("record failure states")
|
||||
|
||||
from unittest.mock import patch as _patch
|
||||
|
||||
from orisys_app_shared.record.experiment import _SessionWriter as _Writer
|
||||
|
||||
scratch_fail = APP_ROOT / "outputs" / "_record_fail"
|
||||
_shutil.rmtree(scratch_fail, ignore_errors=True)
|
||||
scratch_fail.mkdir(parents=True, exist_ok=True)
|
||||
stuck_fail = _StuckWorker()
|
||||
fail_window = StudioMainWindow(
|
||||
prefs={
|
||||
"language": "zh_CN",
|
||||
"theme": "dark",
|
||||
"sensor_type": "box",
|
||||
"source_mode": "camera",
|
||||
}
|
||||
)
|
||||
fail_window.resize(1440, 880)
|
||||
fail_window.show()
|
||||
fail_window._ensure_workers = lambda: None
|
||||
fail_window._box_worker = stuck_fail
|
||||
fail_window._finger_worker = stuck_fail
|
||||
fail_window._start_run()
|
||||
fail_window._run_timer.stop()
|
||||
fail_window._set_phase("running")
|
||||
|
||||
# A directory that cannot be created is a terminal, retryable failure.
|
||||
blocked = scratch_fail / "blocked"
|
||||
blocked.write_text("", encoding="utf-8")
|
||||
_app_module.recordings_dir = lambda: blocked
|
||||
fail_window.toolbar.btn_record.setChecked(True)
|
||||
app.processEvents()
|
||||
check(
|
||||
not fail_window.recorder.is_recording,
|
||||
"a recording started even though its directory could not be created",
|
||||
)
|
||||
check(
|
||||
not fail_window.toolbar.btn_record.isChecked(),
|
||||
"the record button stayed active after the start failed",
|
||||
)
|
||||
check(
|
||||
fail_window.recorder.state == "failed",
|
||||
f"a failed start left the recorder in {fail_window.recorder.state!r}",
|
||||
)
|
||||
|
||||
# ...and the operator can simply try again.
|
||||
_app_module.recordings_dir = lambda: scratch_fail
|
||||
fail_window.toolbar.btn_record.setChecked(True)
|
||||
app.processEvents()
|
||||
check(
|
||||
fail_window.recorder.is_recording,
|
||||
"the recorder refused to retry after a failed directory",
|
||||
)
|
||||
|
||||
# A writer that keeps failing must fail the recording, and the window must
|
||||
# consume that report even though ``is_recording`` is already False.
|
||||
with _patch.object(_Writer, "write_sample", side_effect=OSError("disk full")):
|
||||
for index in range(8):
|
||||
stuck_fail.emit(_fake_packet(index))
|
||||
fail_window._on_run_tick()
|
||||
deadline = time.monotonic() + 3.0
|
||||
while (
|
||||
fail_window.recorder.stats().write_errors < 5
|
||||
and time.monotonic() < deadline
|
||||
):
|
||||
app.processEvents()
|
||||
time.sleep(0.01)
|
||||
check(
|
||||
fail_window.recorder.stats().failed,
|
||||
"repeated write failures never failed the recording",
|
||||
)
|
||||
check(
|
||||
not fail_window.recorder.is_recording,
|
||||
"a dead writer still claimed to be recording",
|
||||
)
|
||||
fail_window._stop_recording()
|
||||
app.processEvents()
|
||||
failed_report = fail_window._record_last_report
|
||||
check(
|
||||
failed_report is not None and failed_report.stats.failed,
|
||||
"the window did not consume the failure report",
|
||||
)
|
||||
check(
|
||||
not fail_window.toolbar.btn_record.isChecked(),
|
||||
"the record button stayed active after the recording failed",
|
||||
)
|
||||
check(
|
||||
fail_window._session_phase == "error",
|
||||
f"the session phase is {fail_window._session_phase!r} after a failed recording",
|
||||
)
|
||||
|
||||
fail_window.close()
|
||||
app.processEvents()
|
||||
stuck_fail.release()
|
||||
_app_module.recordings_dir = lambda: scratch
|
||||
_shutil.rmtree(scratch_fail, ignore_errors=True)
|
||||
|
||||
section("calibration record identity")
|
||||
|
||||
import json as _cal_json
|
||||
from unittest.mock import patch as _patch_mismatch
|
||||
|
||||
import orisys_studio.ui.cal_panel as _cal_panel_module
|
||||
import orisys_studio.calibration_service as _cal_service_module
|
||||
from PyQt5.QtWidgets import QMessageBox as _QMessageBox
|
||||
|
||||
from orisys_app_shared.force.calibration_io import (
|
||||
build_force_calibration_v2 as _build_v2b,
|
||||
write_force_calibration as _write_cal,
|
||||
)
|
||||
from orisys_app_shared.force.calibration_session import (
|
||||
build_session_document as _build_session_doc,
|
||||
)
|
||||
|
||||
record_dir = APP_ROOT / "outputs" / "_cal_record_check"
|
||||
if record_dir.exists():
|
||||
for stale in record_dir.iterdir():
|
||||
stale.unlink()
|
||||
record_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
identity_window = StudioMainWindow(
|
||||
prefs={
|
||||
"language": "zh_CN",
|
||||
"theme": "dark",
|
||||
"sensor_type": "box",
|
||||
"source_mode": "camera",
|
||||
}
|
||||
)
|
||||
identity_window.resize(1400, 880)
|
||||
identity_window.show()
|
||||
identity_window.cal_panel.set_running(True)
|
||||
taken_points = []
|
||||
for load, raw in ((1.0, 100.0), (3.0, 300.0)):
|
||||
identity_window.cal_recorder.start(
|
||||
channel="fn", reference_load_n=load, duration_s=1.0
|
||||
)
|
||||
for index in range(20):
|
||||
identity_window.cal_recorder.feed(
|
||||
_calibration_sample(index, fn_raw=raw)
|
||||
)
|
||||
taken_points.append(identity_window.cal_recorder.finish_now())
|
||||
device_a = record_dir / "device_a_session.json"
|
||||
device_a.write_text(
|
||||
_cal_json.dumps(
|
||||
_build_session_doc(
|
||||
taken_points,
|
||||
meta={
|
||||
"sensor_type": "box",
|
||||
"sn": "SN-A",
|
||||
"config_ref": "D:/lab/deviceA/sensor.json",
|
||||
},
|
||||
)
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
identity_window.cal_recorder.clear_points()
|
||||
|
||||
# The live device is B; opening A's record must not restamp it.
|
||||
panel = identity_window.cal_panel
|
||||
panel.set_meta_context(sensor_type="finger", config_ref="D:/lab/deviceB/sensor.json")
|
||||
panel.sn_edit.setText("SN-B")
|
||||
panel._on_open_record(str(device_a))
|
||||
app.processEvents()
|
||||
check(
|
||||
panel.sn_edit.text() == "SN-A",
|
||||
f"the reopened record's SN was overwritten: {panel.sn_edit.text()!r}",
|
||||
)
|
||||
meta = panel._export_meta()
|
||||
check(meta["sensor_type"] == "box", "the record's sensor type was overwritten")
|
||||
check(meta["sn"] == "SN-A", "the record's SN was not kept")
|
||||
check(
|
||||
meta["config_ref"] == "D:/lab/deviceA/sensor.json",
|
||||
f"the record's config was overwritten: {meta['config_ref']!r}",
|
||||
)
|
||||
check(
|
||||
str(meta.get("source_record", "")).endswith(device_a.name),
|
||||
"the export does not reference the source record",
|
||||
)
|
||||
saved = sorted(record_dir.glob("calibration_session_box_*.json"))
|
||||
with _patch_mismatch.object(
|
||||
_cal_panel_module, "calibration_export_dir", lambda: record_dir
|
||||
):
|
||||
with _patch_mismatch.object(
|
||||
_cal_service_module, "calibration_export_dir", lambda: record_dir
|
||||
):
|
||||
with _patch_mismatch.object(
|
||||
_cal_panel_module.QMessageBox, "information", lambda *a, **k: None
|
||||
):
|
||||
panel._on_save_record()
|
||||
saved = sorted(record_dir.glob("calibration_session_box_*.json"))
|
||||
check(bool(saved), "saving a reopened record produced no file")
|
||||
if saved:
|
||||
saved_meta = _cal_json.loads(saved[-1].read_text(encoding="utf-8"))["meta"]
|
||||
check(
|
||||
saved_meta.get("sn") == "SN-A" and saved_meta.get("sensor_type") == "box",
|
||||
f"the saved record was restamped: {saved_meta.get('sn')!r}",
|
||||
)
|
||||
doc, _warnings = _build_v2b(panel._prepared)
|
||||
check(
|
||||
doc["meta"]["sn"] == "SN-A" and doc["meta"]["sensor_type"] == "box",
|
||||
"the exported coefficients carry the wrong device identity",
|
||||
)
|
||||
|
||||
# A reopened point with only its samples can still be inspected.
|
||||
reopened = identity_window.cal_recorder.points[0]
|
||||
for entry in _build_session_doc([reopened])["points"]:
|
||||
entry["trace"] = {}
|
||||
dialog = panel.point_dialog(reopened)
|
||||
check(dialog is not None, "a reopened point could not be inspected")
|
||||
if dialog is not None:
|
||||
check(
|
||||
len(dialog.plot_widget.ci.items) == 3,
|
||||
"the rebuilt diagnostic is not one trace per axis",
|
||||
)
|
||||
dialog.close()
|
||||
|
||||
# Applying a calibration that names another device is flagged and defaults
|
||||
# to "no".
|
||||
stuck_identity = _StuckWorker()
|
||||
identity_window._ensure_workers = lambda: None
|
||||
identity_window._box_worker = stuck_identity
|
||||
identity_window._finger_worker = stuck_identity
|
||||
coefficients = record_dir / "device_a_coefficients.json"
|
||||
_write_cal(doc, coefficients)
|
||||
check(
|
||||
bool(
|
||||
identity_window.calibration_target_mismatch(
|
||||
str(coefficients),
|
||||
sensor_type="finger",
|
||||
config_ref="D:/lab/B/sensor.json",
|
||||
)
|
||||
),
|
||||
"a calibration for another device was not reported as a mismatch",
|
||||
)
|
||||
check(
|
||||
not identity_window.calibration_target_mismatch(
|
||||
str(coefficients),
|
||||
sensor_type="box",
|
||||
config_ref="D:/lab/deviceA/sensor.json",
|
||||
),
|
||||
"matching identities were reported as a mismatch",
|
||||
)
|
||||
with _patch_mismatch.object(
|
||||
_QMessageBox, "question", staticmethod(lambda *a, **k: _QMessageBox.No)
|
||||
):
|
||||
identity_window._on_calibration_apply(str(coefficients))
|
||||
check(
|
||||
not stuck_identity.calibration_requests,
|
||||
"a mismatched calibration was applied after the user declined",
|
||||
)
|
||||
check(
|
||||
identity_window.statusBar().currentMessage() == t("cal.apply_mismatch_skipped"),
|
||||
f"the refusal was not reported: {identity_window.statusBar().currentMessage()!r}",
|
||||
)
|
||||
# A matching calibration still goes through the normal confirmation.
|
||||
identity_window.sensor_picker.set_sensor_type("box")
|
||||
with _patch_mismatch.object(
|
||||
_QMessageBox, "question", staticmethod(lambda *a, **k: _QMessageBox.Yes)
|
||||
):
|
||||
identity_window._on_calibration_apply(str(coefficients))
|
||||
check(
|
||||
stuck_identity.calibration_requests == [str(coefficients)],
|
||||
f"a matching calibration was not requested: {stuck_identity.calibration_requests}",
|
||||
)
|
||||
|
||||
identity_window.close()
|
||||
app.processEvents()
|
||||
_shutil.rmtree(record_dir, ignore_errors=True)
|
||||
|
||||
section("display units and time base")
|
||||
import math as _math
|
||||
|
||||
stuck_units = _StuckWorker()
|
||||
units_window = StudioMainWindow(
|
||||
prefs={
|
||||
"language": "zh_CN",
|
||||
"theme": "dark",
|
||||
"sensor_type": "box",
|
||||
"source_mode": "camera",
|
||||
}
|
||||
)
|
||||
units_window.resize(1440, 880)
|
||||
units_window.show()
|
||||
units_window._ensure_workers = lambda: None
|
||||
units_window._box_worker = stuck_units
|
||||
units_window._finger_worker = stuck_units
|
||||
units_window._start_run()
|
||||
units_window._run_timer.stop()
|
||||
units_window._set_phase("running")
|
||||
units_plots = units_window.force_plots
|
||||
units_window.toolbar.units_combo.setCurrentIndex(0) # newton
|
||||
units_plots.reset()
|
||||
wall = 1_700_000_000.0
|
||||
for index in range(90):
|
||||
stuck_units.emit(
|
||||
_fake_packet(index, timestamp=wall + index / 30.0, mono_t=index / 30.0)
|
||||
)
|
||||
units_window._on_run_tick()
|
||||
before = units_plots.history.state()
|
||||
end_before = float(before.times[-1]) if before is not None else -1.0
|
||||
check(
|
||||
before is not None and end_before < 60.0,
|
||||
f"the live X axis is not session-relative (ends at {end_before})",
|
||||
)
|
||||
for units_index in (1, 0, 1, 0):
|
||||
units_window.toolbar.units_combo.setCurrentIndex(units_index)
|
||||
app.processEvents()
|
||||
after = units_plots.history.state()
|
||||
check(
|
||||
after is not None and abs(float(after.times[-1]) - end_before) < 1.0,
|
||||
"the X axis jumped across a raw/N switch: "
|
||||
f"{end_before} -> {None if after is None else after.times[-1]}",
|
||||
)
|
||||
check(
|
||||
units_plots.history.sample_count > 0,
|
||||
"the rebuilt history lost every sample",
|
||||
)
|
||||
units_window.close()
|
||||
units_window._box_worker = None
|
||||
stuck_units.release()
|
||||
app.processEvents()
|
||||
|
||||
# A device with only its normal-force axis calibrated: FN in newtons, and the
|
||||
# other two axes explicitly unavailable instead of showing raw counts.
|
||||
stuck_one = _StuckWorker()
|
||||
one_window = StudioMainWindow(
|
||||
prefs={
|
||||
"language": "zh_CN",
|
||||
"theme": "dark",
|
||||
"sensor_type": "box",
|
||||
"source_mode": "camera",
|
||||
}
|
||||
)
|
||||
one_window.resize(1440, 880)
|
||||
one_window.show()
|
||||
one_window._ensure_workers = lambda: None
|
||||
one_window._box_worker = stuck_one
|
||||
one_window._finger_worker = stuck_one
|
||||
one_window._start_run()
|
||||
one_window._run_timer.stop()
|
||||
one_window._set_phase("running")
|
||||
one_window.toolbar.units_combo.setCurrentIndex(0) # newton
|
||||
one_window.force_plots.reset()
|
||||
for index in range(10):
|
||||
stuck_one.emit(
|
||||
_fake_packet(index, calibrated_axes=("fn",), shear_n=float("nan"))
|
||||
)
|
||||
one_window._on_run_tick()
|
||||
app.processEvents()
|
||||
one_plots = one_window.force_plots
|
||||
check(
|
||||
one_plots._available == (True, False, False),
|
||||
f"per-axis availability is wrong: {one_plots._available}",
|
||||
)
|
||||
check(
|
||||
one_plots.value_labels[0].text() != t("chart.value_unavailable"),
|
||||
"the calibrated axis was reported as unavailable",
|
||||
)
|
||||
check(
|
||||
all(
|
||||
one_plots.value_labels[index].text() == t("chart.value_unavailable")
|
||||
for index in (1, 2)
|
||||
),
|
||||
"an axis without a curve showed a number in newton mode",
|
||||
)
|
||||
one_state = one_plots.history.state()
|
||||
check(
|
||||
one_state is not None
|
||||
and _math.isfinite(float(one_state.values[:, 0].max())),
|
||||
"the calibrated axis was not plotted",
|
||||
)
|
||||
check(
|
||||
one_state is not None
|
||||
and not any(_math.isfinite(v) for v in one_state.values[:, 1]),
|
||||
"FX carried a value without a calibration curve",
|
||||
)
|
||||
one_window.close()
|
||||
one_window._box_worker = None
|
||||
stuck_one.release()
|
||||
app.processEvents()
|
||||
|
||||
section("session browsing")
|
||||
import numpy as np
|
||||
|
||||
@@ -1190,6 +1650,31 @@ def main() -> int:
|
||||
all(int(curve.getData()[0].size) == 40 for curve in playback.curves),
|
||||
"replay did not draw every force sample",
|
||||
)
|
||||
|
||||
# Long-history browsing: an interval read is bounded by the interval, and the
|
||||
# whole-session envelope is a band rather than every row.
|
||||
from orisys_app_shared.force.source import RecordingCurveSource
|
||||
from orisys_app_shared.record.index import RecordingIndex
|
||||
from orisys_app_shared.record.playback import RecordingReader as _RecordingReader
|
||||
|
||||
reader = _RecordingReader(recording, max_rows=10)
|
||||
source = RecordingCurveSource(reader)
|
||||
check(source.kind == "disk", "the replay source is not the recording")
|
||||
index = reader.index()
|
||||
check(index.rows == 40, f"the index found {index.rows} rows, not 40")
|
||||
start, stop = index.row_range(0.5, 1.0)
|
||||
check(
|
||||
0 <= start < stop <= index.rows and (stop - start) <= 20,
|
||||
f"an interval read spans {stop - start} rows",
|
||||
)
|
||||
interval = source.window(0.5, 1.0, unit="N")
|
||||
check(0 < len(interval) <= 20, f"the interval read returned {len(interval)} rows")
|
||||
envelope = source.envelope(buckets=8, unit="N")
|
||||
check(0 < len(envelope) <= 8, f"the envelope has {len(envelope)} points")
|
||||
check(
|
||||
reader.info().total_rows == 40,
|
||||
f"the recording reports {reader.info().total_rows} rows",
|
||||
)
|
||||
check(playback.slider.isEnabled(), "the replay position slider is disabled")
|
||||
check(
|
||||
t("playback.warn_depth_recomputed").split("{")[0].strip()[:4]
|
||||
@@ -1268,6 +1753,92 @@ def main() -> int:
|
||||
|
||||
playback.close()
|
||||
app.processEvents()
|
||||
|
||||
# The plan must state where to start, which reset it is anchored to and
|
||||
# whether the state is really recoverable; and a request must belong to the
|
||||
# recording it started in.
|
||||
from orisys_app_shared.record.depth import (
|
||||
STATE_APPROXIMATE,
|
||||
STATE_PREROLL,
|
||||
DepthRecomputer,
|
||||
)
|
||||
from orisys_app_shared.record.playback import RecordingReader
|
||||
|
||||
plan_reader = RecordingReader(recording)
|
||||
planner = DepthRecomputer(plan_reader)
|
||||
frames = plan_reader.info().frames
|
||||
check(bool(frames), "the replay recording has no frame index")
|
||||
plan = planner.plan_for(frames[10])
|
||||
check(
|
||||
plan.state == STATE_APPROXIMATE and plan.anchor == "segment_start",
|
||||
f"a recording with no reset/pre-roll is not reported as approximate: {plan.state}",
|
||||
)
|
||||
check(
|
||||
plan.steps and plan.steps[0].source == plan_reader.segment_path(frames[10].segment),
|
||||
"the plan does not name the video to read",
|
||||
)
|
||||
check(
|
||||
"state_approximate" in plan.warnings,
|
||||
f"the approximate replay is not reported: {plan.warnings}",
|
||||
)
|
||||
# A pre-roll makes the start of a recording reproducible.
|
||||
session_path = recording / "session.json"
|
||||
session_doc = _json.loads(session_path.read_text(encoding="utf-8"))
|
||||
prerecord = recording / "preroll_0001.mp4"
|
||||
prerecord.write_bytes(b"")
|
||||
session_doc["preroll_frames"] = 30
|
||||
session_doc["preroll_segment"] = prerecord.name
|
||||
session_path.write_text(_json.dumps(session_doc), encoding="utf-8")
|
||||
planner.close()
|
||||
planner = DepthRecomputer(RecordingReader(recording))
|
||||
plan = planner.plan_for(planner.reader.info().frames[5])
|
||||
check(
|
||||
plan.state == STATE_PREROLL and plan.preroll == prerecord,
|
||||
f"the pre-roll was not used for the start of the recording: {plan.state}",
|
||||
)
|
||||
check(plan.frames_to_read == 30 + 6, f"unexpected replay cost: {plan.frames_to_read}")
|
||||
planner.close()
|
||||
|
||||
# A reset boundary is re-applied, and a request is bound to its recording.
|
||||
_Blocking.released.clear()
|
||||
second = PlaybackDialog(theme="dark", directory=recording)
|
||||
second.show()
|
||||
app.processEvents()
|
||||
serial_before = second._record_serial
|
||||
blocking = _Blocking()
|
||||
second._recomputer = blocking
|
||||
second._on_recompute_depth()
|
||||
check(second._depth_pending_record == serial_before, "the request has no recording id")
|
||||
check(
|
||||
second._depth_pending_seq is not None,
|
||||
"the request carries no sequence number",
|
||||
)
|
||||
# Opening another recording invalidates the running request...
|
||||
second.open_recording(recording)
|
||||
check(
|
||||
second._record_serial == serial_before + 1,
|
||||
"opening a recording did not invalidate the running request",
|
||||
)
|
||||
check(
|
||||
second._recomputer is not blocking,
|
||||
"the invalidated processor was not replaced",
|
||||
)
|
||||
_Blocking.released.set()
|
||||
for _ in range(40):
|
||||
app.processEvents()
|
||||
_time.sleep(0.05)
|
||||
if not second._depth_busy():
|
||||
break
|
||||
second._poll_depth()
|
||||
app.processEvents()
|
||||
pixmap = second.depth_label.pixmap()
|
||||
check(
|
||||
pixmap is None or pixmap.isNull(),
|
||||
"an answer for a previous recording was drawn into the new one",
|
||||
)
|
||||
second.close()
|
||||
app.processEvents()
|
||||
|
||||
_shutil.rmtree(replay_root, ignore_errors=True)
|
||||
|
||||
print(f"\nscenes: {scene_count} checks: {CHECKS} failures: {len(FAILURES)}")
|
||||
|
||||
@@ -14,20 +14,25 @@ from orisys_app_shared.acquisition.constants import (
|
||||
LOWPASS_CUTOFF_HZ,
|
||||
)
|
||||
from orisys_app_shared.acquisition.sinks import RecordSinkMixin
|
||||
# The packet and the geometry come from ``models``, so the finger worker does not
|
||||
# depend on the box worker; only genuinely shared helpers are imported from it.
|
||||
from orisys_app_shared.acquisition.models import (
|
||||
FixedCameraMode,
|
||||
SdkPacket,
|
||||
ensure_bgr as _ensure_bgr,
|
||||
is_file_source as _is_file_source,
|
||||
load_fixed_mode,
|
||||
orisys_importable,
|
||||
)
|
||||
from orisys_app_shared.acquisition.sdk_worker import (
|
||||
CAMERA_DISCONNECT_TIMEOUT_S,
|
||||
HANDOVER_TIMEOUT_S,
|
||||
STOP_JOIN_TIMEOUT_S,
|
||||
VIDEO_EOF_TIMEOUT_S,
|
||||
FixedCameraMode,
|
||||
SdkPacket,
|
||||
_ensure_bgr,
|
||||
_is_file_source,
|
||||
calibration_digest,
|
||||
load_fixed_mode,
|
||||
loaded_calibration_version,
|
||||
loaded_path,
|
||||
orisys_importable,
|
||||
process_chain,
|
||||
read_forces_from_sensor,
|
||||
)
|
||||
|
||||
@@ -65,13 +70,17 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
*,
|
||||
on_error: Optional[Callable[[str], None]] = None,
|
||||
on_status: Optional[Callable[[str], None]] = None,
|
||||
on_calibration_applied: Optional[Callable[[str, str, int], None]] = None,
|
||||
on_calibration_applied: Optional[Callable[[str, str, int, float, int], None]] = None,
|
||||
on_reset_applied: Optional[Callable[[int, float, int], None]] = None,
|
||||
):
|
||||
self._init_sinks()
|
||||
self._on_error = on_error
|
||||
self._on_status = on_status
|
||||
#: Called from the acquisition thread once new coefficients are loaded.
|
||||
#: Called from the acquisition thread once new coefficients are loaded,
|
||||
#: with ``(path, version, frame_id, mono_t, session_id)``.
|
||||
self._on_calibration_applied = on_calibration_applied
|
||||
#: Called from the acquisition thread once a queued zeroing took effect.
|
||||
self._on_reset_applied = on_reset_applied
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
self._stop = threading.Event()
|
||||
self._lock = threading.Lock()
|
||||
@@ -86,6 +95,8 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
self._end_reason: Optional[str] = None
|
||||
self._phase: str = "idle"
|
||||
self._pending_calibration: Optional[str] = None
|
||||
#: A zeroing requested from another thread, applied by this one.
|
||||
self._pending_reset: bool = False
|
||||
self._mono_t0: float = 0.0
|
||||
self._calibration_version: str = ""
|
||||
|
||||
@@ -129,6 +140,11 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
self._latest = None
|
||||
self._end_reason = None
|
||||
self._phase = "connecting"
|
||||
# This session starts with no calibration of its own: an uncalibrated
|
||||
# runtime must not inherit the previous session's version, and a stale
|
||||
# zeroing request must not fire into the new session's frames.
|
||||
self._calibration_version = ""
|
||||
self._pending_reset = False
|
||||
self._session_id += 1
|
||||
session = self._session_id
|
||||
self._stop.clear()
|
||||
@@ -171,22 +187,54 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
self._phase = "idle"
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Request a tracker reset, applied by the acquisition thread.
|
||||
|
||||
The runtime's tactile reset from the GUI thread would race the frame the
|
||||
loop is processing, so the zeroing happens at a frame boundary on the
|
||||
acquisition thread, which reports the frame and time it took effect.
|
||||
"""
|
||||
with self._lock:
|
||||
runtime = self._runtimes.get(self._session_id)
|
||||
if runtime is None:
|
||||
if self._session_id <= 0:
|
||||
raise RuntimeError("Finger runtime is not running")
|
||||
runtime.tactile.reset()
|
||||
self._pending_reset = True
|
||||
|
||||
def _consume_pending_reset(self, runtime, frame_id: int, mono_t: float) -> None:
|
||||
"""Apply a queued reset and report the frame it actually took effect at."""
|
||||
with self._lock:
|
||||
pending = self._pending_reset
|
||||
self._pending_reset = False
|
||||
session = self._session_id
|
||||
if not pending or runtime is None:
|
||||
return
|
||||
tactile = getattr(runtime, "tactile", None)
|
||||
target = tactile if callable(getattr(tactile, "reset", None)) else None
|
||||
if target is None:
|
||||
sensor = getattr(tactile, "raw_sensor", None)
|
||||
target = sensor if callable(getattr(sensor, "reset", None)) else None
|
||||
if target is None:
|
||||
raise RuntimeError("runtime does not support reset")
|
||||
target.reset()
|
||||
if self._on_reset_applied is not None:
|
||||
self._on_reset_applied(int(frame_id), float(mono_t), int(session))
|
||||
|
||||
def set_pending_calibration(self, path: str | Path | None) -> None:
|
||||
"""Queue a force_calibration.json path for the acquisition thread."""
|
||||
with self._lock:
|
||||
self._pending_calibration = str(path) if path is not None else None
|
||||
|
||||
def _consume_pending_calibration(self, sensor, frame_id: int = 0) -> None:
|
||||
"""Apply queued coefficients and report confirmation, not intent."""
|
||||
def _consume_pending_calibration(
|
||||
self, sensor, frame_id: int = 0, mono_t: float = 0.0
|
||||
) -> None:
|
||||
"""Apply queued coefficients and report confirmation, not intent.
|
||||
|
||||
A rejected document leaves the version alone on purpose: the sensor still
|
||||
has the coefficients it had. What never happens is inventing a version for
|
||||
a device that has no calibration — that is the empty string.
|
||||
"""
|
||||
with self._lock:
|
||||
path = self._pending_calibration
|
||||
self._pending_calibration = None
|
||||
session = self._session_id
|
||||
if path is None or sensor is None:
|
||||
return
|
||||
if not hasattr(sensor, "set_calibration"):
|
||||
@@ -195,7 +243,9 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
version = calibration_digest(path)
|
||||
self._calibration_version = version
|
||||
if self._on_calibration_applied is not None:
|
||||
self._on_calibration_applied(str(path), version, int(frame_id))
|
||||
self._on_calibration_applied(
|
||||
str(path), version, int(frame_id), float(mono_t), int(session)
|
||||
)
|
||||
|
||||
def _set_end_reason(self, reason: str) -> None:
|
||||
with self._lock:
|
||||
@@ -287,11 +337,12 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
# name those coefficients from the first frame, not the request path.
|
||||
tactile = getattr(runtime, "tactile", None)
|
||||
loaded = loaded_calibration_version(tactile) if tactile else ""
|
||||
if loaded:
|
||||
# Assigned even when empty: an uncalibrated device reports *no*
|
||||
# calibration, never the previous session's version.
|
||||
self._calibration_version = loaded
|
||||
if self._on_calibration_applied is not None:
|
||||
if loaded and self._on_calibration_applied is not None:
|
||||
self._on_calibration_applied(
|
||||
str(loaded_path(tactile) or ""), loaded, 0
|
||||
str(loaded_path(tactile) or ""), loaded, 0, 0.0, int(session)
|
||||
)
|
||||
except Exception as exc:
|
||||
self._set_end_reason("error")
|
||||
@@ -341,20 +392,26 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
none_streak_start = None
|
||||
tactile = runtime.tactile
|
||||
compute_error = ""
|
||||
# The frame's acquisition time, so an event that happens on this
|
||||
# frame is dated when it happened rather than when it was queued.
|
||||
frame_mono = max(0.0, time.perf_counter() - self._mono_t0)
|
||||
try:
|
||||
self._consume_pending_calibration(tactile, frame_id)
|
||||
self._consume_pending_calibration(tactile, frame_id, frame_mono)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
message = f"calibration rejected: {type(exc).__name__}: {exc}"
|
||||
if self._on_error is not None and session == self._session_id:
|
||||
self._on_error(message)
|
||||
compute_error = message
|
||||
try:
|
||||
tactile.compute_deformation(decompose=True, check_motion=False)
|
||||
tactile.compute_contact()
|
||||
except Exception as exc: # noqa: BLE001 - reported on the packet
|
||||
compute_error = compute_error or (
|
||||
f"compute: {type(exc).__name__}: {exc}"
|
||||
)
|
||||
# Applied between two frames on this thread, then reported.
|
||||
self._consume_pending_reset(runtime, frame_id, frame_mono)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
message = f"reset failed: {type(exc).__name__}: {exc}"
|
||||
if self._on_error is not None and session == self._session_id:
|
||||
self._on_error(message)
|
||||
# The same chain the offline depth replay runs on the tactile.
|
||||
chain_depth, chain_error = process_chain(tactile)
|
||||
compute_error = compute_error or chain_error
|
||||
t1 = time.perf_counter()
|
||||
frame_times.append(t1)
|
||||
cutoff = t1 - 1.0
|
||||
@@ -395,7 +452,11 @@ class FingerAcquisitionWorker(RecordSinkMixin):
|
||||
stale_forces=bool(compute_error),
|
||||
raw_available=bool(forces.get("raw_available", True)),
|
||||
flow=forces["flow"],
|
||||
depth=forces.get("depth"),
|
||||
depth=(
|
||||
forces.get("depth")
|
||||
if forces.get("depth") is not None
|
||||
else chain_depth
|
||||
),
|
||||
valid_mask=runtime.valid_mask,
|
||||
)
|
||||
with self._lock:
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
"""The data shapes shared by the acquisition workers, the record channel and the UI.
|
||||
|
||||
They live here rather than in one of the workers so that neither worker has to
|
||||
import the other, and so the record channel can describe a packet without
|
||||
importing a device loop. The workers re-export these names for compatibility.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FixedCameraMode:
|
||||
"""The camera geometry a session runs with."""
|
||||
|
||||
width: int
|
||||
height: int
|
||||
fps: float
|
||||
config_ref: str
|
||||
|
||||
def label(self) -> str:
|
||||
return f"{self.width}x{self.height} @{self.fps:g}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class SdkPacket:
|
||||
"""One acquired frame: the images, the force readout and its provenance."""
|
||||
|
||||
raw_frame: np.ndarray
|
||||
processed_frame: np.ndarray
|
||||
timestamp: float
|
||||
fps: float
|
||||
process_ms: float
|
||||
normal_force: float = 0.0
|
||||
shear_x: float = 0.0
|
||||
shear_y: float = 0.0
|
||||
normal_force_raw: float = 0.0
|
||||
shear_x_raw: float = 0.0
|
||||
shear_y_raw: float = 0.0
|
||||
force_unit: str = "raw"
|
||||
force_calibrated: bool = False
|
||||
#: Which axes actually have a calibration curve loaded. The sensor-level
|
||||
#: ``force_calibrated`` flag cannot express a partially calibrated device.
|
||||
calibrated_axes: tuple[str, ...] = ()
|
||||
session_id: int = 0
|
||||
#: Monotonic per-session frame index; the join key for video, CSV and
|
||||
#: calibration records.
|
||||
frame_id: int = 0
|
||||
#: Monotonic, session-relative acquisition time. Used for ordering so a wall
|
||||
#: clock that steps backwards cannot reorder history. ``None`` means the
|
||||
#: producer did not measure one, and the wall clock is used instead — a
|
||||
#: default of ``0.0`` would have made every frame look simultaneous.
|
||||
mono_t: Optional[float] = None
|
||||
#: Wall clock at hand-off, for metadata only. ``None`` means unmeasured.
|
||||
wall_t: Optional[float] = None
|
||||
#: False when the force readout failed. Invalid packets carry NaN forces and
|
||||
#: must never be averaged into calibration data or written as zeros.
|
||||
valid: bool = True
|
||||
invalid_reason: str = ""
|
||||
#: True when the compute chain failed for this frame. The SDK's force fields
|
||||
#: then still hold the value computed for an earlier frame, so they must not
|
||||
#: be recorded as this frame's measurement.
|
||||
stale_forces: bool = False
|
||||
#: Content hash of the calibration actually in effect when this frame was
|
||||
#: acquired; changes mid-session when new coefficients are applied.
|
||||
calibration_version: str = ""
|
||||
#: False when the sensor exposes no RAW channels; the raw fields are then
|
||||
#: NaN rather than a copy of the calibrated values.
|
||||
raw_available: bool = True
|
||||
flow: Optional[np.ndarray] = None
|
||||
depth: Optional[np.ndarray] = None
|
||||
valid_mask: Optional[np.ndarray] = None
|
||||
|
||||
|
||||
def load_fixed_mode(config_ref: str | Path) -> FixedCameraMode:
|
||||
"""Resolve a config reference (a file path or a product name) to its geometry."""
|
||||
ref = str(config_ref).strip()
|
||||
path = Path(ref)
|
||||
if path.is_file():
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
resolved = str(path.resolve())
|
||||
else:
|
||||
import orisys
|
||||
|
||||
data = orisys.configs.load_config(ref)
|
||||
resolved = ref
|
||||
camera = data.get("camera") or {}
|
||||
raw = camera.get("resolution_raw") or [1920, 1080]
|
||||
fps = float(camera.get("fps") or 30.0)
|
||||
return FixedCameraMode(
|
||||
width=int(raw[0]),
|
||||
height=int(raw[1]),
|
||||
fps=fps,
|
||||
config_ref=resolved,
|
||||
)
|
||||
|
||||
|
||||
def ensure_bgr(img: np.ndarray) -> np.ndarray:
|
||||
"""Three-channel BGR view of whatever the sensor hands over."""
|
||||
if img.ndim == 2:
|
||||
return cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
|
||||
if img.ndim == 3 and img.shape[2] == 1:
|
||||
return cv2.cvtColor(img[:, :, 0], cv2.COLOR_GRAY2BGR)
|
||||
if img.ndim == 3 and img.shape[2] == 4:
|
||||
return cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)
|
||||
return img
|
||||
|
||||
|
||||
def is_file_source(video_source: str | int) -> bool:
|
||||
"""True when the source is a file (a replay) rather than a live camera."""
|
||||
return isinstance(video_source, str)
|
||||
|
||||
|
||||
def orisys_importable() -> tuple[bool, str]:
|
||||
"""Whether the SDK can be imported, with the reason when it cannot."""
|
||||
try:
|
||||
import orisys # noqa: F401
|
||||
|
||||
return True, getattr(__import__("orisys"), "__version__", "unknown")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return False, str(exc)
|
||||
@@ -19,6 +19,14 @@ from orisys_app_shared.acquisition.constants import (
|
||||
FORCE_SAMPLE_RATE_HZ,
|
||||
LOWPASS_CUTOFF_HZ,
|
||||
)
|
||||
from orisys_app_shared.acquisition.models import (
|
||||
FixedCameraMode,
|
||||
SdkPacket,
|
||||
ensure_bgr as _ensure_bgr,
|
||||
is_file_source as _is_file_source,
|
||||
load_fixed_mode,
|
||||
orisys_importable,
|
||||
)
|
||||
from orisys_app_shared.acquisition.sinks import RecordSinkMixin
|
||||
|
||||
VIDEO_EOF_TIMEOUT_S = 0.5
|
||||
@@ -29,105 +37,22 @@ STOP_JOIN_TIMEOUT_S = 5.0
|
||||
#: previous session that is still winding down before opening the device.
|
||||
HANDOVER_TIMEOUT_S = 60.0
|
||||
|
||||
# Re-exported so ``from ...sdk_worker import SdkPacket`` keeps working; the
|
||||
# definitions live in ``models`` so neither worker depends on the other.
|
||||
__all__ = [
|
||||
"FixedCameraMode",
|
||||
"SdkPacket",
|
||||
"SdkAcquisitionWorker",
|
||||
"calibration_digest",
|
||||
"load_fixed_mode",
|
||||
"loaded_calibration_version",
|
||||
"loaded_path",
|
||||
"orisys_importable",
|
||||
"process_chain",
|
||||
"read_depth",
|
||||
"read_forces_from_sensor",
|
||||
]
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FixedCameraMode:
|
||||
width: int
|
||||
height: int
|
||||
fps: float
|
||||
config_ref: str
|
||||
|
||||
def label(self) -> str:
|
||||
return f"{self.width}x{self.height} @{self.fps:g}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class SdkPacket:
|
||||
raw_frame: np.ndarray
|
||||
processed_frame: np.ndarray
|
||||
timestamp: float
|
||||
fps: float
|
||||
process_ms: float
|
||||
normal_force: float = 0.0
|
||||
shear_x: float = 0.0
|
||||
shear_y: float = 0.0
|
||||
normal_force_raw: float = 0.0
|
||||
shear_x_raw: float = 0.0
|
||||
shear_y_raw: float = 0.0
|
||||
force_unit: str = "raw"
|
||||
force_calibrated: bool = False
|
||||
#: Which axes actually have a calibration curve loaded. The sensor-level
|
||||
#: ``force_calibrated`` flag cannot express a partially calibrated device.
|
||||
calibrated_axes: tuple[str, ...] = ()
|
||||
session_id: int = 0
|
||||
#: Monotonic per-session frame index; the join key for video, CSV and
|
||||
#: calibration records.
|
||||
frame_id: int = 0
|
||||
#: Monotonic, session-relative acquisition time. Used for ordering so a wall
|
||||
#: clock that steps backwards cannot reorder history. ``None`` means the
|
||||
#: producer did not measure one, and the wall clock is used instead — a
|
||||
#: default of ``0.0`` would have made every frame look simultaneous.
|
||||
mono_t: Optional[float] = None
|
||||
#: Wall clock at hand-off, for metadata only. ``None`` means unmeasured.
|
||||
wall_t: Optional[float] = None
|
||||
#: False when the force readout failed. Invalid packets carry NaN forces and
|
||||
#: must never be averaged into calibration data or written as zeros.
|
||||
valid: bool = True
|
||||
invalid_reason: str = ""
|
||||
#: True when the compute chain failed for this frame. The SDK's force fields
|
||||
#: then still hold the value computed for an earlier frame, so they must not
|
||||
#: be recorded as this frame's measurement.
|
||||
stale_forces: bool = False
|
||||
#: Content hash of the calibration actually in effect when this frame was
|
||||
#: acquired; changes mid-session when new coefficients are applied.
|
||||
calibration_version: str = ""
|
||||
#: False when the sensor exposes no RAW channels; the raw fields are then
|
||||
#: NaN rather than a copy of the calibrated values.
|
||||
raw_available: bool = True
|
||||
flow: Optional[np.ndarray] = None
|
||||
depth: Optional[np.ndarray] = None
|
||||
valid_mask: Optional[np.ndarray] = None
|
||||
|
||||
|
||||
def orisys_importable() -> tuple[bool, str]:
|
||||
try:
|
||||
import orisys # noqa: F401
|
||||
|
||||
return True, getattr(__import__("orisys"), "__version__", "unknown")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return False, str(exc)
|
||||
|
||||
|
||||
def load_fixed_mode(config_ref: str | Path) -> FixedCameraMode:
|
||||
ref = str(config_ref).strip()
|
||||
path = Path(ref)
|
||||
if path.is_file():
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
resolved = str(path.resolve())
|
||||
else:
|
||||
import orisys
|
||||
|
||||
data = orisys.configs.load_config(ref)
|
||||
resolved = ref
|
||||
camera = data.get("camera") or {}
|
||||
raw = camera.get("resolution_raw") or [1920, 1080]
|
||||
fps = float(camera.get("fps") or 30.0)
|
||||
return FixedCameraMode(
|
||||
width=int(raw[0]),
|
||||
height=int(raw[1]),
|
||||
fps=fps,
|
||||
config_ref=resolved,
|
||||
)
|
||||
|
||||
|
||||
def _ensure_bgr(img: np.ndarray) -> np.ndarray:
|
||||
if img.ndim == 2:
|
||||
return cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
|
||||
if img.ndim == 3 and img.shape[2] == 1:
|
||||
return cv2.cvtColor(img[:, :, 0], cv2.COLOR_GRAY2BGR)
|
||||
if img.ndim == 3 and img.shape[2] == 4:
|
||||
return cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)
|
||||
return img
|
||||
|
||||
|
||||
def _sensor_force_calibrated(sensor, *, config_ref: str | Path | None = None) -> bool:
|
||||
@@ -147,6 +72,43 @@ def _sensor_force_calibrated(sensor, *, config_ref: str | Path | None = None) ->
|
||||
return False
|
||||
|
||||
|
||||
def process_chain(sensor) -> tuple[Optional[np.ndarray], str]:
|
||||
"""Run the live processing chain for one already-read frame.
|
||||
|
||||
``get_img`` alone is not the pipeline: the SDK's depth map is produced by
|
||||
``compute_deformation`` + ``compute_contact``. The offline depth replay calls
|
||||
this same function, so it cannot drift from what was recorded live.
|
||||
Returns ``(depth, error)``; the depth is ``None`` when the chain failed or the
|
||||
sensor exposes none.
|
||||
"""
|
||||
compute_error = ""
|
||||
compute_deformation = getattr(sensor, "compute_deformation", None)
|
||||
compute_contact = getattr(sensor, "compute_contact", None)
|
||||
if callable(compute_deformation) and callable(compute_contact):
|
||||
# Same calls and arguments as the acquisition loop.
|
||||
try:
|
||||
compute_deformation(decompose=True, check_motion=False)
|
||||
compute_contact()
|
||||
except Exception as exc: # noqa: BLE001 - reported to the caller
|
||||
compute_error = f"compute: {type(exc).__name__}: {exc}"
|
||||
else:
|
||||
compute_error = f"no processing chain on {type(sensor).__name__}"
|
||||
return read_depth(sensor), compute_error
|
||||
|
||||
|
||||
def read_depth(sensor) -> Optional[np.ndarray]:
|
||||
"""The current depth channel of ``sensor``, or ``None`` when unavailable."""
|
||||
try:
|
||||
info = sensor.info
|
||||
if not hasattr(info, "DEPTH"):
|
||||
return None
|
||||
value = sensor.read_info(info.DEPTH)
|
||||
except Exception: # noqa: BLE001 - reported as "no depth for this frame"
|
||||
logger.exception("Could not read the depth channel")
|
||||
return None
|
||||
return None if value is None else np.asarray(value)
|
||||
|
||||
|
||||
def read_forces_from_sensor(sensor, *, config_ref: str | Path | None = None):
|
||||
"""Read one force sample.
|
||||
|
||||
@@ -284,10 +246,6 @@ def _calibrated_axes(sensor) -> tuple[str, ...]:
|
||||
return tuple(axes)
|
||||
|
||||
|
||||
def _is_file_source(video_source: str | int) -> bool:
|
||||
return isinstance(video_source, str)
|
||||
|
||||
|
||||
def calibration_digest(path: str | Path) -> str:
|
||||
"""Content hash of a calibration document — the calibration's identity.
|
||||
|
||||
@@ -352,14 +310,20 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
*,
|
||||
on_error: Optional[Callable[[str], None]] = None,
|
||||
on_status: Optional[Callable[[str], None]] = None,
|
||||
on_calibration_applied: Optional[Callable[[str, str, int], None]] = None,
|
||||
on_calibration_applied: Optional[Callable[[str, str, int, float, int], None]] = None,
|
||||
on_reset_applied: Optional[Callable[[int, float, int], None]] = None,
|
||||
):
|
||||
self._init_sinks()
|
||||
self._on_error = on_error
|
||||
self._on_status = on_status
|
||||
#: Called from the acquisition thread once new coefficients are actually
|
||||
#: loaded, with ``(path, version, frame_id)``. Confirmation, not intent.
|
||||
#: loaded, with ``(path, version, frame_id, mono_t, session_id)``.
|
||||
#: Confirmation, not intent — and it carries the frame and time so the
|
||||
#: recorded event is dated when it happened.
|
||||
self._on_calibration_applied = on_calibration_applied
|
||||
#: Called from the acquisition thread once a queued zeroing took effect,
|
||||
#: with ``(frame_id, mono_t, session_id)``.
|
||||
self._on_reset_applied = on_reset_applied
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
self._stop = threading.Event()
|
||||
self._lock = threading.Lock()
|
||||
@@ -373,6 +337,8 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
self._end_reason: Optional[str] = None
|
||||
self._phase: str = "idle"
|
||||
self._pending_calibration: Optional[str] = None
|
||||
#: A zeroing requested from another thread, applied by this one.
|
||||
self._pending_reset: bool = False
|
||||
#: Monotonic origin for ``SdkPacket.mono_t``, reset per session.
|
||||
self._mono_t0: float = 0.0
|
||||
#: Content hash of the calibration currently in effect.
|
||||
@@ -418,6 +384,11 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
self._latest = None
|
||||
self._end_reason = None
|
||||
self._phase = "connecting"
|
||||
# The previous session's coefficients say nothing about this one: an
|
||||
# uncalibrated device must not inherit a version it never loaded, and
|
||||
# an old reset request must not fire into the new session's frames.
|
||||
self._calibration_version = ""
|
||||
self._pending_reset = False
|
||||
self._session_id += 1
|
||||
session = self._session_id
|
||||
self._stop.clear()
|
||||
@@ -463,28 +434,57 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
self._phase = "idle"
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Request a tracker reset, applied by the acquisition thread.
|
||||
|
||||
``Sensor.reset()`` from the GUI thread would race the frame the loop is
|
||||
processing, so the actual zeroing happens at a frame boundary on the
|
||||
acquisition thread — which then reports the frame and the time it took
|
||||
effect. The confirmation callback is ``on_reset_applied``.
|
||||
"""
|
||||
with self._lock:
|
||||
sensor = self._sensors.get(self._session_id)
|
||||
if sensor is None:
|
||||
if self._session_id <= 0:
|
||||
raise RuntimeError("Sensor is not running")
|
||||
sensor.reset()
|
||||
self._pending_reset = True
|
||||
|
||||
def _consume_pending_reset(self, sensor, frame_id: int, mono_t: float) -> None:
|
||||
"""Apply a queued reset and report the frame it actually took effect at."""
|
||||
with self._lock:
|
||||
pending = self._pending_reset
|
||||
self._pending_reset = False
|
||||
session = self._session_id
|
||||
if not pending or sensor is None:
|
||||
return
|
||||
reset = getattr(sensor, "reset", None)
|
||||
if not callable(reset):
|
||||
raise RuntimeError("sensor does not support reset")
|
||||
reset()
|
||||
if self._on_reset_applied is not None:
|
||||
self._on_reset_applied(int(frame_id), float(mono_t), int(session))
|
||||
|
||||
def set_pending_calibration(self, path: str | Path | None) -> None:
|
||||
"""Queue a force_calibration.json path for the acquisition thread."""
|
||||
with self._lock:
|
||||
self._pending_calibration = str(path) if path is not None else None
|
||||
|
||||
def _consume_pending_calibration(self, sensor, frame_id: int = 0) -> None:
|
||||
def _consume_pending_calibration(
|
||||
self, sensor, frame_id: int = 0, mono_t: float = 0.0
|
||||
) -> None:
|
||||
"""Apply queued coefficients and report **confirmation**, not intent.
|
||||
|
||||
``Sensor.set_calibration`` loads with ``strict=True`` and raises on an
|
||||
invalid document, so success is observable. The old code swallowed that
|
||||
exception, which is why the app logged "calibration applied" before
|
||||
anything had actually been loaded.
|
||||
|
||||
A rejection leaves the version alone on purpose: the sensor still has the
|
||||
coefficients it had, so the previous version is still the true one. What
|
||||
never happens is inventing a version for a device that has no calibration
|
||||
— that is the empty string, set when the session opened.
|
||||
"""
|
||||
with self._lock:
|
||||
path = self._pending_calibration
|
||||
self._pending_calibration = None
|
||||
session = self._session_id
|
||||
if path is None or sensor is None:
|
||||
return
|
||||
if not hasattr(sensor, "set_calibration"):
|
||||
@@ -493,7 +493,9 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
version = calibration_digest(path)
|
||||
self._calibration_version = version
|
||||
if self._on_calibration_applied is not None:
|
||||
self._on_calibration_applied(str(path), version, int(frame_id))
|
||||
self._on_calibration_applied(
|
||||
str(path), version, int(frame_id), float(mono_t), int(session)
|
||||
)
|
||||
|
||||
def _set_end_reason(self, reason: str) -> None:
|
||||
with self._lock:
|
||||
@@ -574,11 +576,12 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
# here. Without this the record would name the path passed on the
|
||||
# command line instead of the calibration that was applied.
|
||||
loaded = loaded_calibration_version(sensor)
|
||||
if loaded:
|
||||
# Assigned even when empty: an uncalibrated device must report *no*
|
||||
# calibration, never the previous session's version.
|
||||
self._calibration_version = loaded
|
||||
if self._on_calibration_applied is not None:
|
||||
if loaded and self._on_calibration_applied is not None:
|
||||
self._on_calibration_applied(
|
||||
str(loaded_path(sensor) or ""), loaded, 0
|
||||
str(loaded_path(sensor) or ""), loaded, 0, 0.0, int(session)
|
||||
)
|
||||
self._status(f"Streaming {mode.label()}")
|
||||
except Exception as exc:
|
||||
@@ -627,23 +630,32 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
|
||||
none_streak_start = None
|
||||
compute_error = ""
|
||||
# The frame's acquisition time, so an event that happens on this
|
||||
# frame is dated when it happened rather than when it was queued.
|
||||
frame_mono = max(0.0, time.perf_counter() - self._mono_t0)
|
||||
try:
|
||||
# Applying new coefficients is reported separately from the
|
||||
# compute chain: a rejected document is an operator error,
|
||||
# not a numerical failure.
|
||||
self._consume_pending_calibration(sensor, frame_id)
|
||||
self._consume_pending_calibration(sensor, frame_id, frame_mono)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
message = f"calibration rejected: {type(exc).__name__}: {exc}"
|
||||
if self._on_error is not None and session == self._session_id:
|
||||
self._on_error(message)
|
||||
compute_error = message
|
||||
try:
|
||||
sensor.compute_deformation(decompose=True, check_motion=False)
|
||||
sensor.compute_contact()
|
||||
except Exception as exc: # noqa: BLE001 - reported on the packet
|
||||
compute_error = compute_error or (
|
||||
f"compute: {type(exc).__name__}: {exc}"
|
||||
)
|
||||
# The zeroing is applied *here*, between two frames, and only
|
||||
# then reported — the same confirmation rule as the
|
||||
# calibration.
|
||||
self._consume_pending_reset(sensor, frame_id, frame_mono)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
message = f"reset failed: {type(exc).__name__}: {exc}"
|
||||
if self._on_error is not None and session == self._session_id:
|
||||
self._on_error(message)
|
||||
# The same chain the offline depth replay runs. The depth it
|
||||
# reads is the packet's depth, so the channel is read once.
|
||||
chain_depth, chain_error = process_chain(sensor)
|
||||
compute_error = compute_error or chain_error
|
||||
t1 = time.perf_counter()
|
||||
frame_times.append(t1)
|
||||
cutoff = t1 - 1.0
|
||||
@@ -683,7 +695,7 @@ class SdkAcquisitionWorker(RecordSinkMixin):
|
||||
stale_forces=bool(compute_error),
|
||||
raw_available=bool(forces.get("raw_available", True)),
|
||||
flow=forces["flow"],
|
||||
depth=forces.get("depth"),
|
||||
depth=forces.get("depth") if forces.get("depth") is not None else chain_depth,
|
||||
)
|
||||
with self._lock:
|
||||
self._latest = packet
|
||||
|
||||
@@ -8,10 +8,10 @@ from __future__ import annotations
|
||||
|
||||
import math
|
||||
from copy import deepcopy
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from dataclasses import asdict, dataclass, field, replace
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, Mapping
|
||||
from typing import Any, Iterable, Literal, Mapping
|
||||
|
||||
from orisys.calibration import (
|
||||
CalSample as SdkCalSample,
|
||||
@@ -95,8 +95,6 @@ def predict_channel(cal, channel: str, raw: float) -> float:
|
||||
triple = [0.0, 0.0, 0.0]
|
||||
triple[_CONVERT_SLOT[channel]] = float(raw)
|
||||
return float(cal.convert(triple[0], triple[1], triple[2])[_CONVERT_SLOT[channel]])
|
||||
|
||||
|
||||
@dataclass
|
||||
class CalSessionStore:
|
||||
"""In-memory calibration points for FN / FX / FY."""
|
||||
@@ -176,12 +174,8 @@ def meta_from_sensor_json(path: str | Path | None) -> dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
def build_force_calibration_v2(
|
||||
store: CalSessionStore,
|
||||
*,
|
||||
meta: dict[str, Any] | None = None,
|
||||
) -> tuple[dict[str, Any], list[str]]:
|
||||
"""Build schema v2 JSON. Returns (document, skipped_channel_warnings)."""
|
||||
def _meta_blocks(meta: dict[str, Any] | None) -> tuple[dict, dict, dict, str]:
|
||||
"""Split a flat export-meta dict into the SDK's identity/pipeline/extra."""
|
||||
meta = dict(meta or {})
|
||||
identity = {
|
||||
"sensor_type": meta.get("sensor_type", ""),
|
||||
@@ -194,9 +188,9 @@ def build_force_calibration_v2(
|
||||
"config_ref": meta.get("config_ref"),
|
||||
}
|
||||
extra = {
|
||||
k: v
|
||||
for k, v in meta.items()
|
||||
if k
|
||||
key: value
|
||||
for key, value in meta.items()
|
||||
if key
|
||||
not in (
|
||||
"sensor_type",
|
||||
"sn",
|
||||
@@ -207,30 +201,255 @@ def build_force_calibration_v2(
|
||||
"samples",
|
||||
)
|
||||
}
|
||||
return identity, pipeline, extra, str(meta.get("origin") or "user")
|
||||
|
||||
|
||||
def aggregate_samples(rows: Iterable[CalSample]) -> list[CalSample]:
|
||||
"""n-weighted merge of repeated reference loads, sorted by load.
|
||||
|
||||
Sampling the same load twice is normal and deliberate; the model has to see
|
||||
the combined mean and the scatter *between* the samples, not whichever
|
||||
sample happened to be last.
|
||||
"""
|
||||
merged: list[CalSample] = []
|
||||
for sample in rows:
|
||||
for index, existing in enumerate(merged):
|
||||
if abs(existing.weight_n - sample.weight_n) <= WEIGHT_EPS:
|
||||
merged[index] = merge_samples(existing, sample)
|
||||
break
|
||||
else:
|
||||
merged.append(sample)
|
||||
merged.sort(key=lambda item: item.weight_n)
|
||||
return merged
|
||||
|
||||
|
||||
def training_samples(points) -> tuple[dict[str, list[CalSample]], list, list]:
|
||||
"""Split points into the training build and the held-out validation set.
|
||||
|
||||
This is the **single** filter: a point that is excluded, invalid or has no
|
||||
reference load never reaches a model, and a validation point is only ever
|
||||
measured against one. Callers must not re-implement it.
|
||||
"""
|
||||
training: list = []
|
||||
validation: list = []
|
||||
for point in points or ():
|
||||
if bool(getattr(point, "excluded", False)):
|
||||
continue
|
||||
try:
|
||||
cal = sdk_build(
|
||||
store.to_sdk_samples(),
|
||||
load = float(point.reference_load_n)
|
||||
raw_mean = float(point.loaded_stats().mean)
|
||||
except (AttributeError, TypeError, ValueError):
|
||||
continue
|
||||
if not math.isfinite(load) or not math.isfinite(raw_mean):
|
||||
continue
|
||||
if bool(getattr(point, "is_validation", False)):
|
||||
validation.append(point)
|
||||
else:
|
||||
training.append(point)
|
||||
|
||||
by_channel: dict[str, list[CalSample]] = {key: [] for key in CHANNEL_KEYS}
|
||||
for point in training:
|
||||
stats = point.loaded_stats()
|
||||
std = float(stats.std)
|
||||
by_channel[str(point.channel)].append(
|
||||
CalSample(
|
||||
weight_n=float(point.reference_load_n),
|
||||
raw_mean=float(stats.mean),
|
||||
n=int(stats.n or 0),
|
||||
std=std if math.isfinite(std) else 0.0,
|
||||
)
|
||||
)
|
||||
return (
|
||||
{key: aggregate_samples(rows) for key, rows in by_channel.items()},
|
||||
training,
|
||||
validation,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PreparedCalibration:
|
||||
"""One preparation of a calibration session: samples -> model -> diagnostics.
|
||||
|
||||
The diagnostics and the exported document come from *this* object, so the
|
||||
numbers on screen describe exactly the model that is written to disk. Two
|
||||
separate preparations (one for the panel, one for the fit) is how the
|
||||
displayed residual and the applied curve drifted apart in the first place.
|
||||
"""
|
||||
|
||||
model: Any = None
|
||||
samples: dict[str, list[CalSample]] = field(default_factory=dict)
|
||||
training: list = field(default_factory=list)
|
||||
validation: list = field(default_factory=list)
|
||||
#: ``id(point)`` -> prediction error at that point (predicted - reference).
|
||||
residuals: dict[int, float] = field(default_factory=dict)
|
||||
#: ``id(point)`` -> absolute error of a held-out point.
|
||||
validation_errors: dict[int, float] = field(default_factory=dict)
|
||||
warnings: list[str] = field(default_factory=list)
|
||||
meta: dict = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return self.model is not None
|
||||
|
||||
def channel_samples(self, key: str) -> list[CalSample]:
|
||||
return list(self.samples.get(key) or [])
|
||||
|
||||
def method(self, channel: str):
|
||||
curve = self.curve(channel)
|
||||
return str(getattr(curve, "method", "") or "") if curve is not None else None
|
||||
|
||||
def breakpoints(self, channel: str) -> int:
|
||||
curve = self.curve(channel)
|
||||
return len(getattr(curve, "points", ()) or ()) if curve is not None else 0
|
||||
|
||||
def curve(self, channel: str):
|
||||
if self.model is None:
|
||||
return None
|
||||
return self.model.get(channel)
|
||||
|
||||
def predict(self, channel: str, raw: float) -> float:
|
||||
if self.model is None:
|
||||
return float("nan")
|
||||
return predict_channel(self.model, channel, raw)
|
||||
|
||||
def points_in_fit(self, channel: str) -> int:
|
||||
return len(self.samples.get(channel) or [])
|
||||
|
||||
def validation_points(self, channel: str) -> int:
|
||||
return sum(1 for point in self.validation if point.channel == channel)
|
||||
|
||||
def max_residual(self, channel: str) -> float:
|
||||
values = [
|
||||
abs(value)
|
||||
for point in self.training
|
||||
if point.channel == channel
|
||||
for value in [self.residuals.get(id(point))]
|
||||
if value is not None and math.isfinite(value)
|
||||
]
|
||||
return max(values) if values else float("nan")
|
||||
|
||||
def max_validation_error(self, channel: str) -> float:
|
||||
values = [
|
||||
value
|
||||
for point in self.validation
|
||||
if point.channel == channel
|
||||
for value in [self.validation_errors.get(id(point))]
|
||||
if value is not None and math.isfinite(value)
|
||||
]
|
||||
return max(values) if values else float("nan")
|
||||
|
||||
def to_document(self) -> dict[str, Any]:
|
||||
"""The schema-v2 document for this prepared model."""
|
||||
if self.model is None:
|
||||
raise ValueError(
|
||||
"No complete channels to build (each needs ≥2 points). "
|
||||
+ "; ".join(self.warnings)
|
||||
)
|
||||
return self.model.to_dict()
|
||||
|
||||
|
||||
def prepare_from_samples(
|
||||
samples: dict[str, list[CalSample]],
|
||||
*,
|
||||
meta: dict[str, Any] | None = None,
|
||||
) -> PreparedCalibration:
|
||||
"""Build the model and its diagnostics from prepared training samples.
|
||||
|
||||
Repeated loads are merged here too, so a caller that hands in raw (unmerged)
|
||||
samples still gets the same model as one that hands in points.
|
||||
"""
|
||||
merged = {
|
||||
key: aggregate_samples(samples.get(key) or []) for key in CHANNEL_KEYS
|
||||
}
|
||||
identity, pipeline, extra, origin = _meta_blocks(meta)
|
||||
warnings: list[str] = []
|
||||
model = None
|
||||
try:
|
||||
model = sdk_build(
|
||||
{key: [sample.to_sdk() for sample in rows] for key, rows in merged.items()},
|
||||
identity=identity,
|
||||
pipeline=pipeline,
|
||||
origin=str(meta.get("origin") or "user"),
|
||||
origin=origin,
|
||||
extra_meta=extra,
|
||||
)
|
||||
except Exception as exc:
|
||||
# Match previous API: raise ValueError with channel warnings when empty.
|
||||
raise ValueError(str(exc)) from exc
|
||||
|
||||
result = sdk_check(cal)
|
||||
warnings = list(result.warnings)
|
||||
except Exception as exc: # noqa: BLE001 - surfaced as warnings + no model
|
||||
warnings.append(str(exc))
|
||||
if model is not None:
|
||||
try:
|
||||
warnings.extend(sdk_check(model).warnings)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
warnings.append(f"check failed: {exc}")
|
||||
for key in CHANNEL_KEYS:
|
||||
if not cal.has(key):
|
||||
rows = store.channel(key)
|
||||
if not model.has(key):
|
||||
warnings.append(_missing_channel_message(key, merged.get(key) or []))
|
||||
return PreparedCalibration(
|
||||
model=model,
|
||||
samples=merged,
|
||||
warnings=warnings,
|
||||
meta=dict(meta or {}),
|
||||
)
|
||||
|
||||
|
||||
def _missing_channel_message(key: str, rows: list[CalSample]) -> str:
|
||||
if rows:
|
||||
msg = f"{key}: need at least 2 points (have {len(rows)})"
|
||||
else:
|
||||
msg = f"{key}: no samples"
|
||||
if msg not in warnings:
|
||||
warnings.append(msg)
|
||||
return cal.to_dict(), warnings
|
||||
return f"{key}: need at least 2 points (have {len(rows)})"
|
||||
return f"{key}: no samples"
|
||||
|
||||
|
||||
def prepare_calibration(
|
||||
points,
|
||||
*,
|
||||
meta: dict[str, Any] | None = None,
|
||||
) -> PreparedCalibration:
|
||||
"""The one entry point: points -> training samples -> model -> diagnostics."""
|
||||
samples, training, validation = training_samples(points)
|
||||
prepared = prepare_from_samples(samples, meta=meta)
|
||||
residuals: dict[int, float] = {}
|
||||
for point in training:
|
||||
predicted = prepared.predict(str(point.channel), float(point.loaded_stats().mean))
|
||||
if math.isfinite(predicted):
|
||||
residuals[id(point)] = predicted - float(point.reference_load_n)
|
||||
validation_errors: dict[int, float] = {}
|
||||
for point in validation:
|
||||
predicted = prepared.predict(str(point.channel), float(point.loaded_stats().mean))
|
||||
if math.isfinite(predicted):
|
||||
validation_errors[id(point)] = abs(predicted - float(point.reference_load_n))
|
||||
return replace(
|
||||
prepared,
|
||||
training=list(training),
|
||||
validation=list(validation),
|
||||
residuals=residuals,
|
||||
validation_errors=validation_errors,
|
||||
)
|
||||
|
||||
|
||||
def build_export_model(samples: dict[str, list[CalSample]]):
|
||||
"""The SDK model for ``samples`` — the same preparation the panel uses."""
|
||||
return prepare_from_samples(samples).model
|
||||
|
||||
|
||||
def build_force_calibration_v2(
|
||||
source,
|
||||
*,
|
||||
meta: dict[str, Any] | None = None,
|
||||
) -> tuple[dict[str, Any], list[str]]:
|
||||
"""Serialise the prepared model. Returns (document, warnings).
|
||||
|
||||
``source`` is a :class:`PreparedCalibration` (the normal path — the document
|
||||
is exactly the model that was diagnosed) or, for older callers, a
|
||||
:class:`CalSessionStore`, which is prepared here with ``meta``.
|
||||
"""
|
||||
prepared = (
|
||||
source
|
||||
if isinstance(source, PreparedCalibration)
|
||||
else prepare_from_samples(
|
||||
{key: source.channel(key) for key in CHANNEL_KEYS}, meta=meta
|
||||
)
|
||||
)
|
||||
if prepared.model is None:
|
||||
# Match the previous API: raise ValueError carrying the channel warnings.
|
||||
raise ValueError("; ".join(prepared.warnings) or "no complete channels")
|
||||
return prepared.to_document(), list(prepared.warnings)
|
||||
|
||||
|
||||
def write_force_calibration(
|
||||
@@ -245,15 +464,18 @@ def write_force_calibration(
|
||||
|
||||
|
||||
def export_force_calibration_v2(
|
||||
store: CalSessionStore,
|
||||
source,
|
||||
*,
|
||||
export_dir: str | Path,
|
||||
sensor_type: str,
|
||||
meta: dict[str, Any] | None = None,
|
||||
local_bundle_dir: str | Path | None = None,
|
||||
) -> tuple[Path, Path | None, list[str]]:
|
||||
"""Write timestamped export and optional local bundle force_calibration.json."""
|
||||
doc, warnings = build_force_calibration_v2(store, meta=meta)
|
||||
"""Write timestamped export and optional local bundle force_calibration.json.
|
||||
|
||||
``source`` is the prepared calibration (preferred) or a session store.
|
||||
"""
|
||||
doc, warnings = build_force_calibration_v2(source, meta=meta)
|
||||
if not doc.get("channels"):
|
||||
raise ValueError(
|
||||
"No complete channels to export (each needs ≥2 points). "
|
||||
|
||||
@@ -23,123 +23,39 @@ from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Callable, Literal, Optional
|
||||
from typing import Callable, Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
from orisys_app_shared.force.session import NA, SessionSample, summarise
|
||||
|
||||
try: # pragma: no cover - the module is a plain data definition
|
||||
from orisys_app_shared.force.calibration_session import SAMPLE_FIELDS
|
||||
except Exception: # noqa: BLE001 - a partial install must not break sampling
|
||||
SAMPLE_FIELDS = (
|
||||
"frame_id",
|
||||
"t_rel_s",
|
||||
"fn_raw",
|
||||
"fx_raw",
|
||||
"fy_raw",
|
||||
"fn_n",
|
||||
"fx_n",
|
||||
"fy_n",
|
||||
"valid",
|
||||
from orisys_app_shared.force.models import (
|
||||
CHANNEL_KEYS,
|
||||
MAX_TRACE_POINTS,
|
||||
NA,
|
||||
SAMPLE_FIELDS,
|
||||
STATUS_CANCELLED,
|
||||
STATUS_FEW_SAMPLES,
|
||||
STATUS_OK,
|
||||
AxisStats,
|
||||
CalibrationPoint,
|
||||
ChannelKey,
|
||||
trace_dt,
|
||||
trace_from,
|
||||
)
|
||||
from orisys_app_shared.force.session import SessionSample, summarise
|
||||
|
||||
ChannelKey = Literal["fn", "fx", "fy"]
|
||||
CHANNEL_KEYS: tuple[ChannelKey, ...] = ("fn", "fx", "fy")
|
||||
|
||||
STATUS_OK = "ok"
|
||||
STATUS_FEW_SAMPLES = "few_samples"
|
||||
STATUS_CANCELLED = "cancelled"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AxisStats:
|
||||
"""Mean / standard deviation / valid count for one axis."""
|
||||
|
||||
mean: float = NA
|
||||
std: float = NA
|
||||
n: int = 0
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {"mean": self.mean, "std": self.std, "n": self.n}
|
||||
|
||||
|
||||
@dataclass
|
||||
class CalibrationPoint:
|
||||
"""One recorded calibration point, with all three axes captured."""
|
||||
|
||||
channel: ChannelKey
|
||||
reference_load_n: float
|
||||
direction: str = "load"
|
||||
round_index: int = 0
|
||||
note: str = ""
|
||||
started_at: str = ""
|
||||
ended_at: str = ""
|
||||
duration_s: float = 0.0
|
||||
requested_duration_s: float = 0.0
|
||||
n_total: int = 0
|
||||
n_valid: int = 0
|
||||
raw: dict[str, AxisStats] = field(default_factory=dict)
|
||||
newtons: dict[str, AxisStats] = field(default_factory=dict)
|
||||
status: str = STATUS_OK
|
||||
#: Downsampled per-axis raw trace, so a row can be inspected after the fact.
|
||||
trace: dict[str, list[float]] = field(default_factory=dict)
|
||||
trace_dt_s: float = 0.0
|
||||
#: **Audit** series: every sample of the point, in
|
||||
#: :data:`~orisys_app_shared.force.calibration_session.SAMPLE_FIELDS` order.
|
||||
#: The trace above is a preview and must never be the only surviving copy.
|
||||
samples: list[tuple] = field(default_factory=list)
|
||||
#: Diagnostics, all operator-controlled. Excluded points leave the fit;
|
||||
#: validation points are held out of it and used to test it.
|
||||
excluded: bool = False
|
||||
is_validation: bool = False
|
||||
#: Filled in by :meth:`CalibrationRecorder.fit`.
|
||||
residual_n: float = NA
|
||||
|
||||
def loaded_stats(self) -> AxisStats:
|
||||
return self.raw.get(self.channel, AxisStats())
|
||||
|
||||
def axis_mean(self, channel: str) -> float:
|
||||
stats = self.raw.get(channel)
|
||||
return stats.mean if stats is not None else NA
|
||||
|
||||
def state_key(self) -> str:
|
||||
if self.excluded:
|
||||
return "cal.state_excluded"
|
||||
if self.is_validation:
|
||||
return "cal.state_validation"
|
||||
return "cal.state_ok"
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {
|
||||
"channel": self.channel,
|
||||
"reference_load_n": self.reference_load_n,
|
||||
# Only the loaded axis has a known reference load; the other two are
|
||||
# intentionally absent rather than zero.
|
||||
"direction": self.direction,
|
||||
"round_index": self.round_index,
|
||||
"note": self.note,
|
||||
"started_at": self.started_at,
|
||||
"ended_at": self.ended_at,
|
||||
"duration_s": self.duration_s,
|
||||
"requested_duration_s": self.requested_duration_s,
|
||||
"n_total": self.n_total,
|
||||
"n_valid": self.n_valid,
|
||||
"status": self.status,
|
||||
"excluded": self.excluded,
|
||||
"is_validation": self.is_validation,
|
||||
"residual_n": self.residual_n,
|
||||
"raw": {key: stats.as_dict() for key, stats in self.raw.items()},
|
||||
"newtons": {key: stats.as_dict() for key, stats in self.newtons.items()},
|
||||
# The audit series is part of the point's identity, not a UI extra:
|
||||
# without it the only surviving copy of the samples is the
|
||||
# downsampled trace.
|
||||
"sample_fields": list(SAMPLE_FIELDS),
|
||||
"n_samples": len(self.samples),
|
||||
"samples": [list(row) for row in self.samples],
|
||||
}
|
||||
# Re-exported: the models live in ``force.models`` so the session file format can
|
||||
# describe a point without importing the sampling state machine.
|
||||
__all__ = [
|
||||
"AxisStats",
|
||||
"CalibrationPoint",
|
||||
"CalibrationRecorder",
|
||||
"FitResult",
|
||||
"SAMPLE_FIELDS",
|
||||
"trace_dt",
|
||||
"trace_from",
|
||||
]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -177,10 +93,6 @@ class FitResult:
|
||||
}
|
||||
|
||||
|
||||
#: Cap on stored trace samples per axis; peaks are preserved when reducing.
|
||||
MAX_TRACE_POINTS = 400
|
||||
|
||||
|
||||
class CalibrationRecorder:
|
||||
"""Owns the calibration sampling lifecycle."""
|
||||
|
||||
@@ -358,8 +270,8 @@ class CalibrationRecorder:
|
||||
raw={"fn": axis_stats(2), "fx": axis_stats(3), "fy": axis_stats(4)},
|
||||
newtons={"fn": axis_stats(5), "fx": axis_stats(6), "fy": axis_stats(7)},
|
||||
status=STATUS_OK if valid_block.shape[0] >= 2 else STATUS_FEW_SAMPLES,
|
||||
trace=_trace_from(valid_block),
|
||||
trace_dt_s=_trace_dt(valid_block),
|
||||
trace=trace_from(valid_block),
|
||||
trace_dt_s=trace_dt(valid_block),
|
||||
# Every sample is kept, including the invalid ones: a point that was
|
||||
# mostly invalid is itself a finding. ``samples`` holds the original
|
||||
# tuples so ``frame_id`` stays an integer.
|
||||
@@ -456,94 +368,71 @@ class CalibrationRecorder:
|
||||
with self._lock:
|
||||
return [point for point in self.points if point.channel == channel]
|
||||
|
||||
def fit(self, channel: ChannelKey) -> FitResult:
|
||||
def prepared(self, *, meta: Optional[dict] = None):
|
||||
"""The session prepared into one model plus its diagnostics.
|
||||
|
||||
Everything that shows or writes a calibration goes through this, so the
|
||||
panel, the fit summary and the exported file cannot disagree about which
|
||||
points were used.
|
||||
"""
|
||||
from orisys_app_shared.force.calibration_io import prepare_calibration
|
||||
|
||||
with self._lock:
|
||||
points = list(self.points)
|
||||
return prepare_calibration(points, meta=meta)
|
||||
|
||||
def fit(self, channel: ChannelKey, prepared=None) -> FitResult:
|
||||
"""Diagnose the model that export and application would actually use.
|
||||
|
||||
The residual and validation figures used to come from a private
|
||||
least-squares line while the exported file carried a piecewise curve, so
|
||||
the numbers on screen did not describe the calibration that would be
|
||||
applied. Both now come from the exported curve. Excluded points are
|
||||
dropped, validation points are held out of the build and only measured.
|
||||
applied. Both now come from the one prepared model (see
|
||||
:func:`~orisys_app_shared.force.calibration_io.prepare_calibration`):
|
||||
excluded points are dropped, validation points are held out of the build
|
||||
and only measured, and repeated loads are merged before the build.
|
||||
"""
|
||||
from orisys_app_shared.force.calibration_io import (
|
||||
CalSample,
|
||||
build_export_model,
|
||||
predict_channel,
|
||||
)
|
||||
if prepared is None:
|
||||
prepared = self.prepared()
|
||||
|
||||
points = self.points_for(channel)
|
||||
usable = [
|
||||
point
|
||||
for point in points
|
||||
if not point.excluded and not point.is_validation
|
||||
and np.isfinite(point.reference_load_n)
|
||||
and np.isfinite(point.loaded_stats().mean)
|
||||
]
|
||||
for point in points:
|
||||
point.residual_n = NA
|
||||
if len(usable) < 2:
|
||||
return FitResult(channel=channel, n_points=len(usable))
|
||||
value = prepared.residuals.get(id(point))
|
||||
point.residual_n = NA if value is None else float(value)
|
||||
|
||||
samples: dict[str, list[CalSample]] = {key: [] for key in CHANNEL_KEYS}
|
||||
for point in usable:
|
||||
samples[point.channel].append(
|
||||
CalSample(
|
||||
weight_n=float(point.reference_load_n),
|
||||
raw_mean=float(point.loaded_stats().mean),
|
||||
n=int(point.loaded_stats().n),
|
||||
std=float(point.loaded_stats().std)
|
||||
if np.isfinite(point.loaded_stats().std)
|
||||
else 0.0,
|
||||
curve = prepared.curve(channel)
|
||||
n_points = prepared.points_in_fit(channel)
|
||||
if curve is None or n_points < 2:
|
||||
return FitResult(channel=channel, n_points=n_points)
|
||||
|
||||
raws = np.asarray(
|
||||
[
|
||||
float(point.loaded_stats().mean)
|
||||
for point in prepared.training
|
||||
if point.channel == channel
|
||||
],
|
||||
dtype=float,
|
||||
)
|
||||
loads = np.asarray(
|
||||
[
|
||||
float(point.reference_load_n)
|
||||
for point in prepared.training
|
||||
if point.channel == channel
|
||||
],
|
||||
dtype=float,
|
||||
)
|
||||
try:
|
||||
model = build_export_model(samples)
|
||||
curve = model.get(channel)
|
||||
except Exception:
|
||||
# Without a buildable model there is nothing honest to report.
|
||||
return FitResult(channel=channel, n_points=len(usable))
|
||||
if curve is None:
|
||||
return FitResult(channel=channel, n_points=len(usable))
|
||||
|
||||
residuals = []
|
||||
for point in usable:
|
||||
predicted = predict_channel(model, point.channel, point.loaded_stats().mean)
|
||||
residual = predicted - float(point.reference_load_n)
|
||||
point.residual_n = float(residual)
|
||||
residuals.append(residual)
|
||||
|
||||
validation = [
|
||||
point
|
||||
for point in points
|
||||
if not point.excluded and point.is_validation
|
||||
and np.isfinite(point.reference_load_n)
|
||||
and np.isfinite(point.loaded_stats().mean)
|
||||
]
|
||||
validation_error = NA
|
||||
if validation:
|
||||
errors = [
|
||||
abs(
|
||||
predict_channel(model, point.channel, point.loaded_stats().mean)
|
||||
- float(point.reference_load_n)
|
||||
)
|
||||
for point in validation
|
||||
]
|
||||
validation_error = float(max(errors))
|
||||
|
||||
raws = np.asarray([point.loaded_stats().mean for point in usable], dtype=float)
|
||||
loads = np.asarray([point.reference_load_n for point in usable], dtype=float)
|
||||
linear_gain = linear_offset = NA
|
||||
if not np.allclose(raws, raws[0]):
|
||||
if raws.size >= 2 and not np.allclose(raws, raws[0]):
|
||||
linear_gain, linear_offset = (float(v) for v in np.polyfit(raws, loads, 1))
|
||||
|
||||
return FitResult(
|
||||
channel=channel,
|
||||
n_points=len(usable),
|
||||
max_residual_n=float(np.max(np.abs(residuals))),
|
||||
validation_error_n=validation_error,
|
||||
validation_points=len(validation),
|
||||
n_points=n_points,
|
||||
max_residual_n=prepared.max_residual(channel),
|
||||
validation_error_n=prepared.max_validation_error(channel),
|
||||
validation_points=prepared.validation_points(channel),
|
||||
method=str(getattr(curve, "method", "") or ""),
|
||||
breakpoints=len(getattr(curve, "points", ()) or ()),
|
||||
breakpoints=prepared.breakpoints(channel),
|
||||
linear_gain=linear_gain,
|
||||
linear_offset=linear_offset,
|
||||
)
|
||||
@@ -554,37 +443,3 @@ class CalibrationRecorder:
|
||||
self._rounds[channel] = index + 1
|
||||
return index
|
||||
|
||||
|
||||
def _trace_from(valid_block: np.ndarray) -> dict[str, list[float]]:
|
||||
"""Downsampled raw trace per axis, bounded by ``MAX_TRACE_POINTS``."""
|
||||
if valid_block.size == 0:
|
||||
return {}
|
||||
traces: dict[str, list[float]] = {}
|
||||
for key, column in (("fn", 2), ("fx", 3), ("fy", 4)):
|
||||
values = valid_block[:, column]
|
||||
values = values[np.isfinite(values)]
|
||||
if values.size == 0:
|
||||
continue
|
||||
if values.size > MAX_TRACE_POINTS:
|
||||
# Keep the shape of the trace, including its extremes.
|
||||
edges = np.linspace(0, values.size, MAX_TRACE_POINTS + 1, dtype=int)
|
||||
reduced = [
|
||||
values[start:max(start + 1, stop)]
|
||||
for start, stop in zip(edges[:-1], edges[1:])
|
||||
]
|
||||
values = np.asarray(
|
||||
[
|
||||
bucket[int(np.argmax(np.abs(bucket)))]
|
||||
for bucket in reduced
|
||||
if bucket.size
|
||||
]
|
||||
)
|
||||
traces[key] = [float(value) for value in values]
|
||||
return traces
|
||||
|
||||
|
||||
def _trace_dt(valid_block: np.ndarray) -> float:
|
||||
if valid_block.shape[0] < 2:
|
||||
return 0.0
|
||||
span = float(valid_block[-1, 1] - valid_block[0, 1])
|
||||
return span / max(1, valid_block.shape[0] - 1)
|
||||
|
||||
@@ -23,23 +23,19 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable, Optional
|
||||
|
||||
from orisys_app_shared.force.models import (
|
||||
SAMPLE_FIELDS,
|
||||
AxisStats,
|
||||
CalibrationPoint,
|
||||
trace_dt,
|
||||
trace_from,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Format version of the calibration session document.
|
||||
SESSION_FORMAT_VERSION = "1.0"
|
||||
SESSION_REQUIRED = ("format", "format_version", "created_at", "points")
|
||||
#: Columns of each sample row inside a point, in order.
|
||||
SAMPLE_FIELDS = (
|
||||
"frame_id",
|
||||
"t_rel_s",
|
||||
"fn_raw",
|
||||
"fx_raw",
|
||||
"fy_raw",
|
||||
"fn_n",
|
||||
"fx_n",
|
||||
"fy_n",
|
||||
"valid",
|
||||
)
|
||||
POINT_REQUIRED = ("channel", "reference_load_n")
|
||||
|
||||
|
||||
@@ -59,6 +55,29 @@ class CalibrationSessionRecord:
|
||||
def ok(self) -> bool:
|
||||
return bool(self.points) and not self.warnings
|
||||
|
||||
def identity(self) -> dict:
|
||||
"""The device this session belongs to.
|
||||
|
||||
A calibration record is evidence about *one* device: its sensor type, its
|
||||
serial number and the configuration it was taken with. Reopening it must
|
||||
not restamp it with whatever device happens to be connected now.
|
||||
"""
|
||||
return {
|
||||
"sensor_type": str(self.meta.get("sensor_type") or ""),
|
||||
"sn": str(self.meta.get("sn") or ""),
|
||||
"config_ref": str(self.meta.get("config_ref") or ""),
|
||||
"created_at": self.created_at,
|
||||
}
|
||||
|
||||
def describe(self) -> str:
|
||||
identity = self.identity()
|
||||
parts = [identity["sensor_type"] or "?"]
|
||||
if identity["sn"]:
|
||||
parts.append(f"SN {identity['sn']}")
|
||||
if identity["config_ref"]:
|
||||
parts.append(identity["config_ref"])
|
||||
return " / ".join(parts)
|
||||
|
||||
|
||||
def _jsonable(value: Any) -> Any:
|
||||
"""NaN/Inf are not valid JSON; they are written as ``null``."""
|
||||
@@ -147,11 +166,6 @@ def read_calibration_session(path: str | Path) -> CalibrationSessionRecord:
|
||||
if version != SESSION_FORMAT_VERSION:
|
||||
warnings.append(f"format_version {version or '?'} is not {SESSION_FORMAT_VERSION}")
|
||||
|
||||
from orisys_app_shared.force.calibration_recorder import (
|
||||
CalibrationPoint,
|
||||
AxisStats,
|
||||
)
|
||||
|
||||
points = []
|
||||
for index, raw in enumerate(data.get("points") or []):
|
||||
if not isinstance(raw, dict):
|
||||
@@ -228,9 +242,39 @@ def _point_from_dict(raw: dict, CalibrationPoint, AxisStats):
|
||||
values = [float("nan") if value is None else value for value in row]
|
||||
rows.append(tuple(values))
|
||||
point.samples = rows
|
||||
# The full series is the audit data, so everything a diagnostic view needs is
|
||||
# rebuilt from it: the per-axis trace, its sample interval and, when the
|
||||
# stored duration is missing, the real sampling span.
|
||||
restore_diagnostics(point)
|
||||
return point
|
||||
|
||||
|
||||
def restore_diagnostics(point) -> None:
|
||||
"""Rebuild the three-axis trace and the sampling span from the samples."""
|
||||
if not point.samples:
|
||||
return
|
||||
try:
|
||||
import numpy as np
|
||||
except Exception: # noqa: BLE001 - a partial install must still reopen points
|
||||
return
|
||||
|
||||
width = max(len(row) for row in point.samples)
|
||||
padded = [tuple(row) + (float("nan"),) * (width - len(row)) for row in point.samples]
|
||||
block = np.asarray(padded, dtype=np.float64)
|
||||
if block.shape[1] < len(SAMPLE_FIELDS):
|
||||
return
|
||||
valid = block[:, 8] > 0.5
|
||||
valid_block = block[valid]
|
||||
if not point.trace and valid_block.size:
|
||||
point.trace = trace_from(valid_block)
|
||||
if not point.trace_dt_s and valid_block.size:
|
||||
point.trace_dt_s = trace_dt(valid_block)
|
||||
if not point.duration_s and block.shape[0] > 1:
|
||||
span = float(block[-1, 1] - block[0, 1])
|
||||
if span == span and span >= 0.0:
|
||||
point.duration_s = span
|
||||
|
||||
|
||||
def _optional_float(value: Any) -> float:
|
||||
if value is None:
|
||||
return float("nan")
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
"""The calibration data shapes, independent of sampling and of file storage.
|
||||
|
||||
They live here so the sampling state machine and the session file format can both
|
||||
describe a calibration point without importing each other. ``calibration_recorder``
|
||||
and ``calibration_session`` re-export these names for compatibility.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Literal
|
||||
|
||||
import numpy as np
|
||||
|
||||
#: Stands in for "this value is not available".
|
||||
NA = float("nan")
|
||||
|
||||
ChannelKey = Literal["fn", "fx", "fy"]
|
||||
CHANNEL_KEYS: tuple[ChannelKey, ...] = ("fn", "fx", "fy")
|
||||
|
||||
STATUS_OK = "ok"
|
||||
STATUS_FEW_SAMPLES = "few_samples"
|
||||
STATUS_CANCELLED = "cancelled"
|
||||
|
||||
#: Columns of one audit sample row, in order. A calibration record is only
|
||||
#: auditable if this layout is fixed: it is what a reopened point is rebuilt from.
|
||||
SAMPLE_FIELDS = (
|
||||
"frame_id",
|
||||
"t_rel_s",
|
||||
"fn_raw",
|
||||
"fx_raw",
|
||||
"fy_raw",
|
||||
"fn_n",
|
||||
"fx_n",
|
||||
"fy_n",
|
||||
"valid",
|
||||
)
|
||||
|
||||
#: Index of each sample column, so callers do not hard-code positions.
|
||||
SAMPLE_INDEX = {name: index for index, name in enumerate(SAMPLE_FIELDS)}
|
||||
|
||||
#: Cap on stored trace samples per axis; peaks are preserved when reducing.
|
||||
MAX_TRACE_POINTS = 400
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AxisStats:
|
||||
"""Mean / standard deviation / valid count for one axis."""
|
||||
|
||||
mean: float = NA
|
||||
std: float = NA
|
||||
n: int = 0
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {"mean": self.mean, "std": self.std, "n": self.n}
|
||||
|
||||
|
||||
@dataclass
|
||||
class CalibrationPoint:
|
||||
"""One recorded calibration point, with all three axes captured."""
|
||||
|
||||
channel: ChannelKey
|
||||
reference_load_n: float
|
||||
direction: str = "load"
|
||||
round_index: int = 0
|
||||
note: str = ""
|
||||
started_at: str = ""
|
||||
ended_at: str = ""
|
||||
duration_s: float = 0.0
|
||||
requested_duration_s: float = 0.0
|
||||
n_total: int = 0
|
||||
n_valid: int = 0
|
||||
raw: dict[str, AxisStats] = field(default_factory=dict)
|
||||
newtons: dict[str, AxisStats] = field(default_factory=dict)
|
||||
status: str = STATUS_OK
|
||||
#: Downsampled per-axis raw trace, so a row can be inspected after the fact.
|
||||
trace: dict[str, list[float]] = field(default_factory=dict)
|
||||
trace_dt_s: float = 0.0
|
||||
#: **Audit** series: every sample of the point, in :data:`SAMPLE_FIELDS` order.
|
||||
#: The trace above is a preview and must never be the only surviving copy.
|
||||
samples: list[tuple] = field(default_factory=list)
|
||||
#: Diagnostics, all operator-controlled. Excluded points leave the fit;
|
||||
#: validation points are held out of it and used to test it.
|
||||
excluded: bool = False
|
||||
is_validation: bool = False
|
||||
#: Filled in by :meth:`CalibrationRecorder.fit`.
|
||||
residual_n: float = NA
|
||||
|
||||
def loaded_stats(self) -> AxisStats:
|
||||
return self.raw.get(self.channel, AxisStats())
|
||||
|
||||
def axis_mean(self, channel: str) -> float:
|
||||
stats = self.raw.get(channel)
|
||||
return stats.mean if stats is not None else NA
|
||||
|
||||
def state_key(self) -> str:
|
||||
if self.excluded:
|
||||
return "cal.state_excluded"
|
||||
if self.is_validation:
|
||||
return "cal.state_validation"
|
||||
return "cal.state_ok"
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {
|
||||
"channel": self.channel,
|
||||
"reference_load_n": self.reference_load_n,
|
||||
# Only the loaded axis has a known reference load; the other two are
|
||||
# intentionally absent rather than zero.
|
||||
"direction": self.direction,
|
||||
"round_index": self.round_index,
|
||||
"note": self.note,
|
||||
"started_at": self.started_at,
|
||||
"ended_at": self.ended_at,
|
||||
"duration_s": self.duration_s,
|
||||
"requested_duration_s": self.requested_duration_s,
|
||||
"n_total": self.n_total,
|
||||
"n_valid": self.n_valid,
|
||||
"status": self.status,
|
||||
"excluded": self.excluded,
|
||||
"is_validation": self.is_validation,
|
||||
"residual_n": self.residual_n,
|
||||
"raw": {key: stats.as_dict() for key, stats in self.raw.items()},
|
||||
"newtons": {key: stats.as_dict() for key, stats in self.newtons.items()},
|
||||
# The audit series is part of the point's identity, not a UI extra:
|
||||
# without it the only surviving copy of the samples is the
|
||||
# downsampled trace.
|
||||
"sample_fields": list(SAMPLE_FIELDS),
|
||||
"n_samples": len(self.samples),
|
||||
"samples": [list(row) for row in self.samples],
|
||||
}
|
||||
|
||||
|
||||
def trace_from(valid_block: np.ndarray) -> dict[str, list[float]]:
|
||||
"""Downsampled raw trace per axis, bounded by ``MAX_TRACE_POINTS``.
|
||||
|
||||
One downsampling rule, whether the point was just recorded or read back from
|
||||
a saved session.
|
||||
"""
|
||||
if valid_block.size == 0:
|
||||
return {}
|
||||
traces: dict[str, list[float]] = {}
|
||||
for key, column in (("fn", 2), ("fx", 3), ("fy", 4)):
|
||||
values = valid_block[:, column]
|
||||
values = values[np.isfinite(values)]
|
||||
if values.size == 0:
|
||||
continue
|
||||
if values.size > MAX_TRACE_POINTS:
|
||||
# Keep the shape of the trace, including its extremes.
|
||||
edges = np.linspace(0, values.size, MAX_TRACE_POINTS + 1, dtype=int)
|
||||
reduced = [
|
||||
values[start:max(start + 1, stop)]
|
||||
for start, stop in zip(edges[:-1], edges[1:])
|
||||
]
|
||||
values = np.asarray(
|
||||
[
|
||||
bucket[int(np.argmax(np.abs(bucket)))]
|
||||
for bucket in reduced
|
||||
if bucket.size
|
||||
]
|
||||
)
|
||||
traces[key] = [float(value) for value in values]
|
||||
return traces
|
||||
|
||||
|
||||
def trace_dt(valid_block: np.ndarray) -> float:
|
||||
"""The sample interval of a trace, from the block it was taken from."""
|
||||
if valid_block.shape[0] < 2:
|
||||
return 0.0
|
||||
span = float(valid_block[-1, 1] - valid_block[0, 1])
|
||||
return span / max(1, valid_block.shape[0] - 1)
|
||||
@@ -29,6 +29,8 @@ from typing import Iterable, Iterator, Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
from orisys_app_shared.record.sample import plottable_axis_mask
|
||||
|
||||
#: Stands in for "this value is not available / not valid".
|
||||
NA = float("nan")
|
||||
|
||||
@@ -127,14 +129,20 @@ class SessionSlice:
|
||||
def plottable(self, unit: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""``(times, values, mask)`` for drawing, under one shared rule.
|
||||
|
||||
A point is drawn only when the sample is **valid** *and* the value for the
|
||||
requested unit is finite. Filtering on finiteness alone let a stale
|
||||
number — left behind by a failed compute step — reappear in one view
|
||||
after another view had already excluded it.
|
||||
``mask`` is per axis ``(n, 3)``: a slice is drawn only where the sample is
|
||||
**valid** *and that axis* is finite. Filtering on finiteness alone let a
|
||||
stale number — left behind by a failed compute step — reappear in one view
|
||||
after another view had already excluded it, and an all-axes rule hid a
|
||||
valid FN whenever the device had no shear calibration.
|
||||
"""
|
||||
values = self.value_for_unit(unit)
|
||||
mask = self.valid & np.isfinite(values).all(axis=1)
|
||||
return self.times[mask], values[mask], mask
|
||||
return self.times, values, plottable_axis_mask(values, self.valid)
|
||||
|
||||
def plottable_axis(self, unit: str, index: int) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""``(times, values)`` for one channel, with the other axes ignored."""
|
||||
values = self.value_for_unit(unit)
|
||||
mask = plottable_axis_mask(values, self.valid)[:, index]
|
||||
return self.times[mask], values[mask, index]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
"""One interface for reading curve data, whatever it is stored in.
|
||||
|
||||
The live charts read a bounded in-memory session store; the replay reads a
|
||||
recording on disk. Both have to answer the same questions — what is the extent,
|
||||
what is in this interval, what does the whole session look like as an envelope —
|
||||
so the plotting code asks a *source* instead of knowing which one it has.
|
||||
|
||||
A source never hands over more than the caller's point budget, and it always
|
||||
reports what it had to do to fit: ``decimated`` means the values are a view, not
|
||||
every sample.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
import numpy as np
|
||||
|
||||
from orisys_app_shared.force.session import NA, SessionSlice
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceStats:
|
||||
"""What the selection readout shows, for any source."""
|
||||
|
||||
duration_s: float = 0.0
|
||||
samples: int = 0
|
||||
peaks: tuple[float, float, float] = (NA, NA, NA)
|
||||
means: tuple[float, float, float] = (NA, NA, NA)
|
||||
stds: tuple[float, float, float] = (NA, NA, NA)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SourceWindow:
|
||||
"""Rows for one interval, in the shape the charts already consume."""
|
||||
|
||||
times: np.ndarray
|
||||
values: np.ndarray
|
||||
valid: np.ndarray
|
||||
frame_ids: np.ndarray
|
||||
unit: str = "raw"
|
||||
decimated: bool = False
|
||||
|
||||
def __len__(self) -> int:
|
||||
return int(self.times.size)
|
||||
|
||||
def plottable_axis(self, index: int):
|
||||
from orisys_app_shared.record.sample import plottable_axis_mask
|
||||
|
||||
mask = plottable_axis_mask(self.values, self.valid)[:, index]
|
||||
return self.times[mask], self.values[mask, index]
|
||||
|
||||
def statistics(self) -> SourceStats:
|
||||
mask = np.isfinite(self.values) & self.valid[:, None]
|
||||
peaks: list[float] = []
|
||||
means: list[float] = []
|
||||
stds: list[float] = []
|
||||
for index in range(3):
|
||||
column = self.values[:, index][mask[:, index]]
|
||||
if column.size == 0:
|
||||
peaks.append(NA)
|
||||
means.append(NA)
|
||||
stds.append(NA)
|
||||
continue
|
||||
peaks.append(float(np.nanmax(column)))
|
||||
means.append(float(np.nanmean(column)))
|
||||
stds.append(float(np.nanstd(column)))
|
||||
span = float(self.times[-1] - self.times[0]) if self.times.size > 1 else 0.0
|
||||
return SourceStats(
|
||||
duration_s=span,
|
||||
samples=int(mask.any(axis=1).sum()),
|
||||
peaks=tuple(peaks),
|
||||
means=tuple(means),
|
||||
stds=tuple(stds),
|
||||
)
|
||||
|
||||
|
||||
class CurveSource(Protocol):
|
||||
"""What the charts need from wherever the data lives."""
|
||||
|
||||
kind: str
|
||||
|
||||
def extent(self) -> tuple[float, float]: ...
|
||||
|
||||
def window(self, t0: float, t1: float, *, max_points: int | None = None) -> SourceWindow: ...
|
||||
|
||||
def envelope(self, *, buckets: int = 1200) -> SourceWindow: ...
|
||||
|
||||
def duration_s(self) -> float: ...
|
||||
|
||||
|
||||
class StoreCurveSource:
|
||||
"""The live session store: already in memory, bounded by its own blocks."""
|
||||
|
||||
kind = "memory"
|
||||
|
||||
def __init__(self, store=None) -> None:
|
||||
self.store = store
|
||||
|
||||
def set_store(self, store) -> None:
|
||||
self.store = store
|
||||
|
||||
def extent(self) -> tuple[float, float]:
|
||||
if self.store is None:
|
||||
return (0.0, 0.0)
|
||||
return self.store.extent()
|
||||
|
||||
def duration_s(self) -> float:
|
||||
if self.store is None:
|
||||
return 0.0
|
||||
first, last = self.store.extent()
|
||||
return max(0.0, last - first)
|
||||
|
||||
def _slice(self, t0: float, t1: float, max_points: int | None, unit: str) -> SessionSlice:
|
||||
if self.store is None:
|
||||
empty = np.empty(0)
|
||||
return SessionSlice(
|
||||
times=empty,
|
||||
raw=np.empty((0, 3)),
|
||||
newtons=np.empty((0, 3)),
|
||||
valid=np.empty(0, dtype=bool),
|
||||
frame_ids=np.empty(0, dtype=np.int64),
|
||||
)
|
||||
return self.store.visible(
|
||||
t0, t1, max_points=max_points or 0, unit=unit
|
||||
)
|
||||
|
||||
def window(
|
||||
self,
|
||||
t0: float,
|
||||
t1: float,
|
||||
*,
|
||||
max_points: int | None = None,
|
||||
unit: str = "raw",
|
||||
) -> SourceWindow:
|
||||
slice_ = self._slice(t0, t1, max_points, unit)
|
||||
return SourceWindow(
|
||||
times=slice_.times,
|
||||
values=slice_.value_for_unit(unit),
|
||||
valid=slice_.valid,
|
||||
frame_ids=slice_.frame_ids,
|
||||
unit=unit,
|
||||
decimated=bool(max_points and slice_.times.size >= int(max_points)),
|
||||
)
|
||||
|
||||
def envelope(self, *, buckets: int = 1200, unit: str = "raw") -> SourceWindow:
|
||||
"""The whole session, decimated — the store's own bounded view."""
|
||||
if self.store is None:
|
||||
return self.window(0.0, 0.0)
|
||||
first, last = self.store.extent()
|
||||
return self.window(first, last, max_points=int(buckets) * 2, unit=unit)
|
||||
|
||||
def statistics(self, t0: float, t1: float, *, unit: str = "raw") -> SourceStats:
|
||||
if self.store is None:
|
||||
return SourceStats()
|
||||
stats = self.store.statistics(t0, t1, unit=unit)
|
||||
return SourceStats(
|
||||
duration_s=stats.duration_s,
|
||||
samples=stats.samples,
|
||||
peaks=stats.peaks,
|
||||
means=stats.means,
|
||||
stds=stats.stds,
|
||||
)
|
||||
|
||||
|
||||
class RecordingCurveSource:
|
||||
"""A recording on disk: interval reads plus a cached envelope."""
|
||||
|
||||
kind = "disk"
|
||||
|
||||
def __init__(self, reader) -> None:
|
||||
self.reader = reader
|
||||
|
||||
def extent(self) -> tuple[float, float]:
|
||||
return self.reader.index().extent()
|
||||
|
||||
def duration_s(self) -> float:
|
||||
return self.reader.index().duration_s()
|
||||
|
||||
def window(
|
||||
self,
|
||||
t0: float,
|
||||
t1: float,
|
||||
*,
|
||||
max_points: int | None = None,
|
||||
unit: str = "raw",
|
||||
) -> SourceWindow:
|
||||
window = self.reader.window(t0, t1, unit=unit, max_points=max_points)
|
||||
return SourceWindow(
|
||||
times=window.times,
|
||||
values=window.value_for_unit(unit),
|
||||
valid=window.valid,
|
||||
frame_ids=window.frame_ids,
|
||||
unit=unit,
|
||||
decimated=window.decimated,
|
||||
)
|
||||
|
||||
def envelope(self, *, buckets: int = 1200, unit: str = "raw") -> SourceWindow:
|
||||
"""The whole-session envelope, from the index's cached overview."""
|
||||
overview = self.reader.overview(unit=unit, buckets=int(buckets))
|
||||
keep = ~np.isnan(overview.times)
|
||||
times = overview.times[keep]
|
||||
if times.size == 0:
|
||||
return self.window(0.0, 0.0, unit=unit)
|
||||
# The envelope is drawn as a band: the low and high of each bucket. It is
|
||||
# reported as a plain window so callers need only one shape.
|
||||
values = np.where(
|
||||
np.isnan(overview.highs[keep]), overview.lows[keep], overview.highs[keep]
|
||||
)
|
||||
return SourceWindow(
|
||||
times=times,
|
||||
values=values,
|
||||
valid=np.ones(times.size, dtype=bool),
|
||||
frame_ids=np.full(times.size, -1, dtype=np.int64),
|
||||
unit=unit,
|
||||
decimated=True,
|
||||
)
|
||||
|
||||
def statistics(self, t0: float, t1: float, *, unit: str = "raw") -> SourceStats:
|
||||
stats = self.reader.statistics(t0, t1, unit=unit)
|
||||
return SourceStats(
|
||||
duration_s=stats.duration_s,
|
||||
samples=stats.samples,
|
||||
peaks=stats.peaks,
|
||||
means=stats.means,
|
||||
stds=stats.stds,
|
||||
)
|
||||
@@ -1,46 +1,104 @@
|
||||
"""Force unit selection for display."""
|
||||
"""Force unit selection for display.
|
||||
|
||||
The decision is made **per axis** from the standardised measurement
|
||||
(:class:`~orisys_app_shared.record.sample.RecordSample`), not from a device-level
|
||||
``force_calibrated`` flag. A device can legitimately have only its normal-force
|
||||
axis calibrated: the old rule then either showed every axis in raw (losing the
|
||||
one axis that really was in newtons) or, worse, showed raw counts in the newton
|
||||
panels. An axis without a curve is now simply **unavailable** in newton mode and
|
||||
reported as such.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from orisys_app_shared.record.sample import AXES
|
||||
|
||||
#: Index of each axis inside the triplets, matching ``AXES``.
|
||||
AXIS_INDEX: dict[str, int] = {axis: index for index, axis in enumerate(AXES)}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DisplayForces:
|
||||
fn: float
|
||||
fx: float
|
||||
fy: float
|
||||
"""What the live view should show for one sample."""
|
||||
|
||||
#: One value per axis, in ``AXES`` order. NaN means "not available".
|
||||
values: tuple[float, float, float]
|
||||
#: The unit the values actually are (``N`` only when they are newtons).
|
||||
unit: str
|
||||
#: Per axis: is this value a real measurement in ``unit``?
|
||||
available: tuple[bool, bool, bool]
|
||||
#: The requested unit was newton but no axis had a curve, so raw is shown.
|
||||
using_raw_fallback: bool
|
||||
#: What the operator asked for (``N``/``raw``), for the fallback message.
|
||||
requested_unit: str
|
||||
|
||||
def value(self, axis: str) -> float:
|
||||
return self.values[AXIS_INDEX[axis]]
|
||||
|
||||
def is_available(self, axis: str) -> bool:
|
||||
return self.available[AXIS_INDEX[axis]]
|
||||
|
||||
def unavailable_axes(self) -> tuple[str, ...]:
|
||||
return tuple(
|
||||
axis for axis in AXES if not self.available[AXIS_INDEX[axis]]
|
||||
)
|
||||
|
||||
|
||||
def select_display_forces(
|
||||
*,
|
||||
units_pref: str,
|
||||
force_calibrated: bool,
|
||||
fn: float,
|
||||
fx: float,
|
||||
fy: float,
|
||||
fn_raw: float,
|
||||
fx_raw: float,
|
||||
fy_raw: float,
|
||||
) -> DisplayForces:
|
||||
"""Return values for plots/labels given newton/raw preference."""
|
||||
def _finite(value: float) -> bool:
|
||||
try:
|
||||
return value == value and value not in (float("inf"), float("-inf"))
|
||||
except TypeError: # pragma: no cover - defensive
|
||||
return False
|
||||
|
||||
|
||||
def select_display_forces(*, units_pref: str, sample) -> DisplayForces:
|
||||
"""Pick the values to display for one standardised measurement.
|
||||
|
||||
``sample`` is a
|
||||
:class:`~orisys_app_shared.record.sample.RecordSample`, so the same conversion
|
||||
that feeds the CSV and the session store also decides what the live charts
|
||||
show: there is no second interpretation of what a value means.
|
||||
"""
|
||||
want_newton = str(units_pref or "newton").lower().startswith("n")
|
||||
if want_newton and force_calibrated:
|
||||
return DisplayForces(fn=fn, fx=fx, fy=fy, unit="N", using_raw_fallback=False)
|
||||
if want_newton and not force_calibrated:
|
||||
raw = tuple(sample.raw_for(axis) for axis in AXES)
|
||||
newton = tuple(sample.newton_for(axis) for axis in AXES)
|
||||
requested = "N" if want_newton else "raw"
|
||||
|
||||
if not want_newton:
|
||||
return DisplayForces(
|
||||
fn=fn_raw,
|
||||
fx=fx_raw,
|
||||
fy=fy_raw,
|
||||
unit="raw",
|
||||
using_raw_fallback=True,
|
||||
)
|
||||
return DisplayForces(
|
||||
fn=fn_raw,
|
||||
fx=fx_raw,
|
||||
fy=fy_raw,
|
||||
values=raw,
|
||||
unit="raw",
|
||||
available=tuple(_finite(value) for value in raw),
|
||||
using_raw_fallback=False,
|
||||
requested_unit=requested,
|
||||
)
|
||||
|
||||
values: list[float] = []
|
||||
available: list[bool] = []
|
||||
for index, _axis in enumerate(AXES):
|
||||
if _finite(newton[index]):
|
||||
values.append(newton[index])
|
||||
available.append(True)
|
||||
else:
|
||||
# Not "zero" and not the raw count: there is no newton value at all.
|
||||
values.append(float("nan"))
|
||||
available.append(False)
|
||||
|
||||
if not any(available):
|
||||
# Nothing is calibrated: showing raw is honest as long as it is labelled.
|
||||
return DisplayForces(
|
||||
values=raw,
|
||||
unit="raw",
|
||||
available=tuple(_finite(value) for value in raw),
|
||||
using_raw_fallback=True,
|
||||
requested_unit=requested,
|
||||
)
|
||||
return DisplayForces(
|
||||
values=tuple(values),
|
||||
unit="N",
|
||||
available=tuple(available),
|
||||
using_raw_fallback=False,
|
||||
requested_unit=requested,
|
||||
)
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
"""Experiment recording (video + data + metadata) and replay."""
|
||||
|
||||
from orisys_app_shared.record.calibration_snapshot import (
|
||||
list_calibration_snapshots,
|
||||
store_calibration_snapshot,
|
||||
)
|
||||
from orisys_app_shared.record.depth import DepthRecomputer, DepthResult
|
||||
from orisys_app_shared.record.experiment import (
|
||||
EVENTS_HEADER,
|
||||
@@ -34,5 +38,7 @@ __all__ = [
|
||||
"RecordingInfo",
|
||||
"RecordingReader",
|
||||
"format_value",
|
||||
"list_calibration_snapshots",
|
||||
"list_recordings",
|
||||
"store_calibration_snapshot",
|
||||
]
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Store the coefficients a recording was made with, next to the recording.
|
||||
|
||||
The hash of a calibration tells you *which* coefficients were applied; it does not
|
||||
let you check what they were, and the original export file can be overwritten or
|
||||
deleted afterwards. So the recording keeps its own copy:
|
||||
|
||||
```
|
||||
outputs/recordings/<stamp>_session<NNNN>/
|
||||
calibration/
|
||||
force_calibration_<hash12>.json
|
||||
```
|
||||
|
||||
The copy is content-addressed by the digest, so applying the same calibration
|
||||
twice in one session stores one file, and two different calibrations can never
|
||||
collide.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Directory inside a recording that holds the coefficient snapshots.
|
||||
SNAPSHOT_DIRNAME = "calibration"
|
||||
|
||||
|
||||
def snapshot_name(version: str, source: Path) -> str:
|
||||
"""The snapshot's file name: content-addressed, with the original extension."""
|
||||
digest = str(version or "").strip()
|
||||
stem = f"force_calibration_{digest[:12]}" if digest else f"force_calibration_{source.stem}"
|
||||
return f"{stem}{source.suffix or '.json'}"
|
||||
|
||||
|
||||
def store_calibration_snapshot(directory: str | Path, source: str | Path, version: str) -> str:
|
||||
"""Copy ``source`` into ``directory/calibration``. Returns the relative name.
|
||||
|
||||
Raises on I/O failure; the caller reports it and keeps recording.
|
||||
"""
|
||||
base = Path(directory) / SNAPSHOT_DIRNAME
|
||||
base.mkdir(parents=True, exist_ok=True)
|
||||
origin = Path(source)
|
||||
target = base / snapshot_name(version, origin)
|
||||
if not target.is_file() or target.stat().st_size != origin.stat().st_size:
|
||||
# Same content-addressed name means the same coefficients; a size check
|
||||
# is enough to notice a truncated previous copy.
|
||||
shutil.copy2(origin, target)
|
||||
return f"{SNAPSHOT_DIRNAME}/{target.name}"
|
||||
|
||||
|
||||
def list_calibration_snapshots(directory: str | Path) -> list[str]:
|
||||
"""Every snapshot already stored in a recording, sorted."""
|
||||
base = Path(directory) / SNAPSHOT_DIRNAME
|
||||
if not base.is_dir():
|
||||
return []
|
||||
return sorted(
|
||||
f"{SNAPSHOT_DIRNAME}/{path.name}" for path in base.iterdir() if path.is_file()
|
||||
)
|
||||
@@ -25,6 +25,7 @@ from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
from orisys_app_shared.record.depth_backends import open_replay
|
||||
from orisys_app_shared.record.playback import RecordingReader
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -40,25 +41,72 @@ class DepthResult:
|
||||
t_rel_s: float = 0.0
|
||||
#: What a replay could not reproduce exactly (never silently empty).
|
||||
warnings: tuple[str, ...] = ()
|
||||
#: Which recording this answer belongs to (the replay window compares it).
|
||||
record_id: str = ""
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return self.depth is not None and not self.error
|
||||
|
||||
|
||||
#: The replay started at a **reset**: the algorithm's state was cleared there, so
|
||||
#: the frames since that point determine it exactly.
|
||||
STATE_EXACT = "exact"
|
||||
#: The replay is preceded by the recorded pre-roll: the intended reconstruction.
|
||||
STATE_PREROLL = "preroll"
|
||||
#: Neither: the state entering the target frame is unknown and may differ from
|
||||
#: what the live run had.
|
||||
STATE_APPROXIMATE = "approximate"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReplayStep:
|
||||
"""One leg of a replay: read ``source`` up to ``to_frame``.
|
||||
|
||||
``reset_after`` is the frame at which the algorithm's state was cleared live
|
||||
(an operator ``zeroing``). The replay reaches that frame, then calls the
|
||||
processor's ``reset()`` — which is what the live run did — and keeps reading.
|
||||
"""
|
||||
|
||||
source: Path
|
||||
to_frame: int
|
||||
reset_after: Optional[int] = None
|
||||
|
||||
|
||||
#: How many SDK reads a faithful replay may cost before it is declared
|
||||
#: unrecoverable (5 minutes at 30 fps). A replay that re-reads an hour of video
|
||||
#: to answer one cursor move is not a feature; saying so is.
|
||||
MAX_REPLAY_FRAMES = 9000
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReplayPlan:
|
||||
"""Where the SDK must start reading to reproduce one frame's state."""
|
||||
|
||||
segment: Path
|
||||
#: Frame within ``segment`` to step to (1-based, as the SDK reads them).
|
||||
stop_at: int
|
||||
#: Frames to feed before ``stop_at`` to rebuild the algorithm's state.
|
||||
#: The legs of the replay, in order. The last one ends at ``stop_at``.
|
||||
steps: tuple[ReplayStep, ...] = ()
|
||||
#: Frame within the final segment the caller asked for (1-based).
|
||||
stop_at: int = 0
|
||||
#: Frames to feed from another file before the first step.
|
||||
preroll: Path | None = None
|
||||
preroll_frames: int = 0
|
||||
#: The ``zeroing`` frame the plan is anchored to, when there is one.
|
||||
reset_frame_id: Optional[int] = None
|
||||
#: ``reset`` / ``preroll`` / ``segment_start``.
|
||||
anchor: str = "segment_start"
|
||||
#: :data:`STATE_EXACT`, :data:`STATE_PREROLL` or :data:`STATE_APPROXIMATE`.
|
||||
state: str = STATE_APPROXIMATE
|
||||
warnings: tuple[str, ...] = ()
|
||||
error: str = ""
|
||||
|
||||
@property
|
||||
def frames_to_read(self) -> int:
|
||||
"""How many SDK reads a faithful recomputation costs."""
|
||||
return int(self.preroll_frames) + sum(
|
||||
max(0, int(step.to_frame)) for step in self.steps
|
||||
)
|
||||
|
||||
|
||||
class DepthRecomputer:
|
||||
"""Recompute depth maps for one recording, on demand."""
|
||||
@@ -69,6 +117,12 @@ class DepthRecomputer:
|
||||
self._segment: Optional[Path] = None
|
||||
#: How many frames the open sensor has already consumed in ``_segment``.
|
||||
self._position = 0
|
||||
#: The depth of ``_position``, so asking for the same frame twice is a
|
||||
#: cache hit instead of a second (impossible) read.
|
||||
self._last_depth: Optional[np.ndarray] = None
|
||||
#: True when an earlier frame had to be re-read without a reset boundary
|
||||
#: to clean up after it, so the state may not match the live run.
|
||||
self._rewound_without_anchor = False
|
||||
self.last_error = ""
|
||||
self.last_warnings: list[str] = []
|
||||
|
||||
@@ -121,46 +175,87 @@ class DepthRecomputer:
|
||||
"""Decide where replay must start so ``frame`` is reproducible.
|
||||
|
||||
The SDK's depth is the output of a stateful chain (optical-flow tracker,
|
||||
temporal filters), so a frame cannot be recomputed in isolation. Two
|
||||
things make the state recoverable:
|
||||
temporal filters), so a frame cannot be recomputed in isolation. What the
|
||||
plan states explicitly:
|
||||
|
||||
* an operator **reset** clears the tracker, so replaying from the reset
|
||||
frame reproduces the state exactly — no earlier history needed;
|
||||
* the **pre-roll** recorded with the session holds the frames acquired
|
||||
just before recording started, which is the only history a reset-free
|
||||
start has.
|
||||
* **where** to start (``start_at``, plus a pre-roll file when there is one);
|
||||
* which **reset boundary** the replay is anchored to, if any;
|
||||
* whether the state is actually recoverable — :data:`STATE_EXACT` (the
|
||||
replay starts at a reset), :data:`STATE_PREROLL` (the recorded pre-roll
|
||||
precedes it) or :data:`STATE_APPROXIMATE` (neither: the depth map is
|
||||
still produced, and the caller is told it may differ from live).
|
||||
|
||||
Anything else is replayed as faithfully as possible *and reported*: the
|
||||
caller is told the state may differ instead of being shown a depth map
|
||||
that pretends to be the live one.
|
||||
Finding a reset event is **not** by itself a recovered state: the reset
|
||||
frame has to exist in the index *and* in the video, and it has to be in
|
||||
the same segment, because the SDK cannot carry its state across a segment
|
||||
boundary.
|
||||
"""
|
||||
warnings: list[str] = []
|
||||
segment_path = self.reader.segment_path(frame.segment)
|
||||
if segment_path is None:
|
||||
return ReplayPlan(
|
||||
segment=Path(), stop_at=0, error="segment_missing"
|
||||
)
|
||||
return ReplayPlan(segment=Path(), error="segment_missing")
|
||||
|
||||
stop_at = max(1, int(frame.segment_frame))
|
||||
reset_frame = self.reader.last_reset_frame(frame.frame_id)
|
||||
if reset_frame is not None:
|
||||
origin = self.reader.frame_of(reset_frame)
|
||||
if origin is not None and origin.segment == frame.segment:
|
||||
# Exact: the tracker state at ``frame`` is fully determined by
|
||||
# the frames since the reset in this same segment.
|
||||
return ReplayPlan(
|
||||
if origin is None:
|
||||
warnings.append("reset_frame_not_indexed")
|
||||
elif origin.dropped or not origin.in_video:
|
||||
# The boundary frame is not in the video, so the state at that
|
||||
# point cannot be rebuilt: replay from here is an approximation.
|
||||
warnings.append("reset_frame_not_in_video")
|
||||
elif origin.segment == frame.segment:
|
||||
plan = ReplayPlan(
|
||||
segment=segment_path,
|
||||
stop_at=max(1, int(frame.segment_frame)),
|
||||
warnings=(),
|
||||
steps=(
|
||||
ReplayStep(
|
||||
source=segment_path,
|
||||
to_frame=stop_at,
|
||||
reset_after=int(origin.segment_frame),
|
||||
),
|
||||
),
|
||||
stop_at=stop_at,
|
||||
reset_frame_id=int(reset_frame),
|
||||
anchor="reset",
|
||||
state=STATE_EXACT,
|
||||
)
|
||||
if origin is not None:
|
||||
# The reset is in an earlier segment. The SDK opens one video per
|
||||
# sensor, so the state cannot be carried across the boundary; the
|
||||
# replay restarts at this segment and says so.
|
||||
if plan.frames_to_read > MAX_REPLAY_FRAMES:
|
||||
warnings.append("state_too_far_back")
|
||||
else:
|
||||
return plan
|
||||
else:
|
||||
# The reset is in an earlier segment. The processor survives a
|
||||
# source swap, so the state *can* be carried: read the tail of
|
||||
# the reset's segment, then continue in the target's.
|
||||
reset_segment = self.reader.segment_path(origin.segment)
|
||||
last_of_reset_segment = self._last_segment_frame(origin.segment)
|
||||
if reset_segment is None or last_of_reset_segment is None:
|
||||
warnings.append("state_across_segment")
|
||||
else:
|
||||
plan = ReplayPlan(
|
||||
segment=segment_path,
|
||||
steps=(
|
||||
ReplayStep(
|
||||
source=reset_segment,
|
||||
to_frame=int(last_of_reset_segment),
|
||||
reset_after=int(origin.segment_frame),
|
||||
),
|
||||
ReplayStep(source=segment_path, to_frame=stop_at),
|
||||
),
|
||||
stop_at=stop_at,
|
||||
reset_frame_id=int(reset_frame),
|
||||
anchor="reset",
|
||||
state=STATE_EXACT,
|
||||
)
|
||||
if plan.frames_to_read <= MAX_REPLAY_FRAMES:
|
||||
return plan
|
||||
# Too expensive: report it rather than freeze the window.
|
||||
warnings.append("state_too_far_back")
|
||||
|
||||
start = self.reader.segment_start_frame(frame.segment)
|
||||
if start is None:
|
||||
return ReplayPlan(segment=Path(), stop_at=0, error="frame_not_indexed")
|
||||
return ReplayPlan(segment=Path(), error="frame_not_indexed")
|
||||
|
||||
preroll = self.reader.preroll_segment()
|
||||
preroll_frames = self.reader.preroll_frames() if preroll is not None else 0
|
||||
@@ -169,19 +264,39 @@ class DepthRecomputer:
|
||||
# one are the only history that can rebuild the tracker.
|
||||
return ReplayPlan(
|
||||
segment=segment_path,
|
||||
stop_at=max(1, int(frame.segment_frame)),
|
||||
steps=(ReplayStep(source=segment_path, to_frame=stop_at),),
|
||||
stop_at=stop_at,
|
||||
preroll=preroll,
|
||||
preroll_frames=int(preroll_frames),
|
||||
anchor="preroll",
|
||||
state=STATE_PREROLL,
|
||||
warnings=tuple(warnings),
|
||||
)
|
||||
if int(frame.segment_frame) > 1:
|
||||
# No reset and no pre-roll: the state entering this frame is unknown.
|
||||
warnings.append("state_approximate")
|
||||
return ReplayPlan(
|
||||
segment=segment_path,
|
||||
stop_at=max(1, int(frame.segment_frame)),
|
||||
warnings=tuple(warnings),
|
||||
steps=(
|
||||
ReplayStep(
|
||||
source=segment_path,
|
||||
to_frame=stop_at,
|
||||
),
|
||||
),
|
||||
stop_at=stop_at,
|
||||
anchor="segment_start",
|
||||
state=STATE_APPROXIMATE,
|
||||
warnings=tuple(dict.fromkeys(warnings)),
|
||||
)
|
||||
|
||||
def _last_segment_frame(self, segment: int) -> Optional[int]:
|
||||
"""The highest indexed frame number inside ``segment``."""
|
||||
frames = [
|
||||
int(entry.segment_frame)
|
||||
for entry in self.reader.info().frames
|
||||
if int(entry.segment) == int(segment)
|
||||
]
|
||||
return max(frames) if frames else None
|
||||
|
||||
# ----------------------------------------------------------------- recompute
|
||||
def depth_at(self, t_rel_s: float) -> DepthResult:
|
||||
"""Recompute the depth map for the frame at ``t_rel_s``."""
|
||||
@@ -219,32 +334,95 @@ class DepthRecomputer:
|
||||
logger.exception("Depth recomputation failed")
|
||||
self.last_error = f"{type(exc).__name__}: {exc}"
|
||||
return DepthResult(
|
||||
error=self.last_error, frame_id=frame.frame_id, t_rel_s=frame.t_rel_s
|
||||
error=self.last_error,
|
||||
frame_id=frame.frame_id,
|
||||
t_rel_s=frame.t_rel_s,
|
||||
warnings=plan.warnings,
|
||||
record_id=self.record_id(),
|
||||
)
|
||||
self.last_warnings = list(plan.warnings)
|
||||
if self._rewound_without_anchor and "state_approximate" not in self.last_warnings:
|
||||
# The plan looked recoverable, but the read path had to restart a file
|
||||
# without a reset boundary to clear the tracker afterwards.
|
||||
self.last_warnings.append("state_approximate_after_rewind")
|
||||
warnings = tuple(dict.fromkeys(self.last_warnings))
|
||||
if depth is None:
|
||||
self.last_error = "depth_not_available"
|
||||
return DepthResult(
|
||||
error=self.last_error,
|
||||
frame_id=frame.frame_id,
|
||||
t_rel_s=frame.t_rel_s,
|
||||
warnings=plan.warnings,
|
||||
warnings=warnings,
|
||||
record_id=self.record_id(),
|
||||
)
|
||||
self.last_error = ""
|
||||
return DepthResult(
|
||||
depth=depth,
|
||||
frame_id=frame.frame_id,
|
||||
t_rel_s=frame.t_rel_s,
|
||||
warnings=plan.warnings,
|
||||
warnings=warnings,
|
||||
record_id=self.record_id(),
|
||||
)
|
||||
|
||||
def record_id(self) -> str:
|
||||
"""Which recording these answers belong to."""
|
||||
return str(getattr(self.reader, "directory", "") or "")
|
||||
|
||||
# ------------------------------------------------------------------ playback
|
||||
def _replay(self, plan: ReplayPlan) -> Optional[np.ndarray]:
|
||||
"""Read the pre-roll (if any), then step to the planned frame."""
|
||||
"""Execute the plan's steps, keeping the processor and swapping inputs.
|
||||
|
||||
The **processor is kept** across a source change: the SDK's tracker lives
|
||||
on the object, so disposing of it would dispose of the state the pre-roll
|
||||
(or the reset boundary) just rebuilt. A reset boundary is re-applied by
|
||||
calling ``reset()`` where the operator's ``zeroing`` happened.
|
||||
"""
|
||||
self._rewound_without_anchor = False
|
||||
if plan.preroll is not None and plan.preroll_frames > 0:
|
||||
self._feed(plan.preroll, int(plan.preroll_frames))
|
||||
if self._segment != plan.segment or self._position > plan.stop_at:
|
||||
self._reopen_segment(plan.segment)
|
||||
return self._advance_to(int(plan.stop_at))
|
||||
depth: Optional[np.ndarray] = None
|
||||
for step in plan.steps:
|
||||
depth = self._run_step(plan, step)
|
||||
return depth
|
||||
|
||||
def _run_step(self, plan: ReplayPlan, step: ReplayStep) -> Optional[np.ndarray]:
|
||||
target = max(1, int(step.to_frame))
|
||||
if self._sensor is None or self._segment != step.source:
|
||||
if self._sensor is None or not self._swap_source(step.source):
|
||||
self._open_segment(step.source)
|
||||
elif self._position > target:
|
||||
# Re-reading an earlier frame can only be done from the start of the
|
||||
# file. Without a reset boundary to clean up after it, the tracker has
|
||||
# now seen frames it should not have.
|
||||
if not self._swap_source(step.source):
|
||||
self._open_segment(step.source)
|
||||
if plan.anchor != "reset":
|
||||
self._rewound_without_anchor = True
|
||||
|
||||
last: Optional[np.ndarray] = (
|
||||
self._last_depth if self._position == target else None
|
||||
)
|
||||
while self._position < target:
|
||||
depth = self._read_one()
|
||||
if depth is None:
|
||||
break
|
||||
last = depth
|
||||
if step.reset_after is not None and self._position == int(step.reset_after):
|
||||
# The operator's zeroing: the live run cleared the tracker here.
|
||||
if not self._reset_processor():
|
||||
self.last_warnings.append("reset_not_applied")
|
||||
return last
|
||||
|
||||
def _reset_processor(self) -> bool:
|
||||
replay = self._sensor
|
||||
reset = getattr(replay, "reset", None)
|
||||
if not callable(reset):
|
||||
return False
|
||||
try:
|
||||
return bool(reset())
|
||||
except Exception: # noqa: BLE001 - reported as a replay warning
|
||||
logger.exception("Could not re-apply the reset boundary")
|
||||
return False
|
||||
|
||||
def _depth_from_segment(
|
||||
self, segment: Path, segment_frame: int
|
||||
@@ -253,28 +431,55 @@ class DepthRecomputer:
|
||||
|
||||
The SDK's video source only advances, so stepping costs one read per
|
||||
frame — and the position is tracked, so asking for frame 2 and then
|
||||
frame 3 reads two frames and then one, instead of two more.
|
||||
frame 3 reads two frames and then one, instead of two more. Asking twice
|
||||
for the *same* frame is answered from the cached depth of that frame.
|
||||
"""
|
||||
target = max(1, int(segment_frame))
|
||||
if self._segment != segment or self._position > target:
|
||||
self._reopen_segment(segment)
|
||||
return self._advance_to(target)
|
||||
if self._segment != segment:
|
||||
self._open_segment(segment)
|
||||
return self._run_step(
|
||||
ReplayPlan(segment=Path(segment), stop_at=target),
|
||||
ReplayStep(source=Path(segment), to_frame=target),
|
||||
)
|
||||
|
||||
# --------------------------------------------------------------------- video
|
||||
def sensor_type(self) -> str:
|
||||
device = self.reader.session().get("device") or {}
|
||||
return str(device.get("sensor_type") or "box")
|
||||
|
||||
def _reopen_segment(self, segment: Path) -> None:
|
||||
def _open_segment(self, segment: Path) -> None:
|
||||
"""Open a segment on a **new** processor (the previous one is closed)."""
|
||||
self._close()
|
||||
if self.sensor_type() == "finger":
|
||||
self._sensor = _FingerReplay(
|
||||
segment, self.resolved_config(), self._finger_export_dir()
|
||||
self._sensor = open_replay(
|
||||
self.sensor_type(),
|
||||
Path(segment),
|
||||
self.resolved_config(),
|
||||
self._finger_export_dir(),
|
||||
)
|
||||
else:
|
||||
self._sensor = _BoxReplay(segment, self.resolved_config())
|
||||
self._segment = Path(segment)
|
||||
self._position = 0
|
||||
self._last_depth = None
|
||||
|
||||
def _swap_source(self, segment: Path) -> bool:
|
||||
"""Replace only the video source of the open processor.
|
||||
|
||||
Returns False when the backend cannot do it, in which case the caller
|
||||
falls back to a full reopen (and the algorithm state is lost, which the
|
||||
plan already reports as approximate).
|
||||
"""
|
||||
replay = self._sensor
|
||||
if replay is None:
|
||||
return False
|
||||
try:
|
||||
if not replay.set_source(Path(segment)):
|
||||
return False
|
||||
except Exception: # noqa: BLE001 - reported by the plan's warnings
|
||||
logger.exception("Could not swap the replayed video source")
|
||||
return False
|
||||
self._segment = Path(segment)
|
||||
self._position = 0
|
||||
self._last_depth = None
|
||||
return True
|
||||
|
||||
def _finger_export_dir(self):
|
||||
"""The finger pipeline assets, resolved exactly as the live run did."""
|
||||
@@ -292,29 +497,17 @@ class DepthRecomputer:
|
||||
"""Consume ``frames`` frames from another file to rebuild state."""
|
||||
if frames <= 0:
|
||||
return
|
||||
if self._sensor is None or self._segment != segment:
|
||||
self._reopen_segment(segment)
|
||||
if self._sensor is None:
|
||||
self._open_segment(segment)
|
||||
elif self._segment != segment:
|
||||
if not self._swap_source(segment):
|
||||
self._open_segment(segment)
|
||||
for _ in range(int(frames)):
|
||||
if self._read_one() is None:
|
||||
break
|
||||
|
||||
def _advance_to(self, target: int) -> Optional[np.ndarray]:
|
||||
last_depth = None
|
||||
while self._position < int(target):
|
||||
depth = self._read_one()
|
||||
if depth is None:
|
||||
break
|
||||
last_depth = depth
|
||||
return last_depth
|
||||
|
||||
def _read_one(self) -> Optional[np.ndarray]:
|
||||
"""Read one frame through the **live** processing chain.
|
||||
|
||||
``get_img``/``read_source`` alone is not the live pipeline: the SDK's
|
||||
depth map is produced by ``compute_deformation`` + ``compute_contact``
|
||||
(and, for the finger, the unrolling step), so skipping them would replay
|
||||
a different algorithm than the one that produced the recording.
|
||||
"""
|
||||
"""Read one frame through the **live** processing chain (see ``process_chain``)."""
|
||||
replay = self._sensor
|
||||
if replay is None:
|
||||
return None
|
||||
@@ -322,12 +515,14 @@ class DepthRecomputer:
|
||||
if depth is None:
|
||||
return None
|
||||
self._position += 1
|
||||
self._last_depth = depth
|
||||
return depth
|
||||
|
||||
def _close(self) -> None:
|
||||
replay, self._sensor = self._sensor, None
|
||||
self._segment = None
|
||||
self._position = 0
|
||||
self._last_depth = None
|
||||
if replay is None:
|
||||
return
|
||||
try:
|
||||
@@ -343,97 +538,3 @@ class DepthRecomputer:
|
||||
|
||||
def __exit__(self, *_exc) -> None:
|
||||
self.close()
|
||||
|
||||
|
||||
class _BoxReplay:
|
||||
"""The box sensor's live chain, opened on one video file."""
|
||||
|
||||
def __init__(self, segment: Path, config_ref: str) -> None:
|
||||
import orisys
|
||||
|
||||
self._sensor = orisys.Sensor(
|
||||
str(segment),
|
||||
config_name=config_ref,
|
||||
verbose=False,
|
||||
backend="auto",
|
||||
)
|
||||
|
||||
def read(self) -> Optional[np.ndarray]:
|
||||
image = self._sensor.get_img()
|
||||
if image is None:
|
||||
return None
|
||||
compute_deformation = getattr(self._sensor, "compute_deformation", None)
|
||||
compute_contact = getattr(self._sensor, "compute_contact", None)
|
||||
if callable(compute_deformation) and callable(compute_contact):
|
||||
# Same calls and arguments as the acquisition loop.
|
||||
compute_deformation(decompose=True, check_motion=False)
|
||||
compute_contact()
|
||||
else:
|
||||
logger.warning(
|
||||
"Sensor %r has no processing chain; replay depth is not the live algorithm",
|
||||
type(self._sensor).__name__,
|
||||
)
|
||||
info = self._sensor.info
|
||||
if not hasattr(info, "DEPTH"):
|
||||
return None
|
||||
value = self._sensor.read_info(info.DEPTH)
|
||||
return None if value is None else np.asarray(value)
|
||||
|
||||
def close(self) -> None:
|
||||
self._sensor.disconnect()
|
||||
|
||||
|
||||
class _FingerReplay:
|
||||
"""The finger's live chain: unroll the source frame, then process the tactile."""
|
||||
|
||||
def __init__(self, segment: Path, config_ref: str, export_dir) -> None:
|
||||
from orisys.finger import FingerRuntime
|
||||
|
||||
from orisys_app_shared.acquisition.constants import (
|
||||
FORCE_SAMPLE_RATE_HZ,
|
||||
LOWPASS_CUTOFF_HZ,
|
||||
)
|
||||
|
||||
kwargs: dict = {
|
||||
"config": config_ref,
|
||||
"video": str(segment),
|
||||
"verbose": False,
|
||||
}
|
||||
if export_dir is not None:
|
||||
kwargs["export_dir"] = str(export_dir)
|
||||
self._runtime = FingerRuntime.open(**kwargs)
|
||||
self._runtime.warmup(color=False)
|
||||
tactile = getattr(self._runtime, "tactile", None)
|
||||
if tactile is not None and hasattr(tactile, "configure"):
|
||||
try:
|
||||
# The same filter settings the acquisition loop applies.
|
||||
tactile.configure(
|
||||
lowpass_cutoff_hz=LOWPASS_CUTOFF_HZ,
|
||||
lowpass_sample_rate_hz=FORCE_SAMPLE_RATE_HZ,
|
||||
)
|
||||
except Exception: # noqa: BLE001 - the defaults still work
|
||||
logger.exception("Could not configure the replayed tactile sensor")
|
||||
|
||||
def read(self) -> Optional[np.ndarray]:
|
||||
ok, source_frame = self._runtime.read_source()
|
||||
if not ok or source_frame is None:
|
||||
return None
|
||||
unrolled = self._runtime.process_frame(source_frame, color=False)
|
||||
if unrolled is None:
|
||||
return None
|
||||
tactile = self._runtime.tactile
|
||||
compute_deformation = getattr(tactile, "compute_deformation", None)
|
||||
compute_contact = getattr(tactile, "compute_contact", None)
|
||||
if callable(compute_deformation) and callable(compute_contact):
|
||||
compute_deformation(decompose=True, check_motion=False)
|
||||
compute_contact()
|
||||
else:
|
||||
logger.warning("Finger tactile sensor has no processing chain")
|
||||
info = tactile.info
|
||||
if not hasattr(info, "DEPTH"):
|
||||
return None
|
||||
value = tactile.read_info(info.DEPTH)
|
||||
return None if value is None else np.asarray(value)
|
||||
|
||||
def close(self) -> None:
|
||||
self._runtime.close()
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Replay backends: the live processing chain, pointed at a video file.
|
||||
|
||||
Each backend keeps **one** processor for its lifetime. The tracker state lives on
|
||||
that object, so a replay must be able to change the *input source* without
|
||||
throwing the processor away — that is what makes a pre-roll, or a zeroing that
|
||||
happened in an earlier segment, reproducible at all.
|
||||
|
||||
The processing itself is not duplicated here: the reads go through
|
||||
``acquisition.sdk_worker.process_chain``, the same function the live workers run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class BoxReplay:
|
||||
"""The box sensor's live chain, opened on one video file."""
|
||||
|
||||
def __init__(self, segment: Path, config_ref: str) -> None:
|
||||
import orisys
|
||||
|
||||
self._config_ref = config_ref
|
||||
self._sensor = orisys.Sensor(
|
||||
str(segment),
|
||||
config_name=config_ref,
|
||||
verbose=False,
|
||||
backend="auto",
|
||||
)
|
||||
|
||||
def set_source(self, segment: Path) -> bool:
|
||||
"""Point the same processor at another file.
|
||||
|
||||
The tracker lives on the ``Sensor`` object, so re-opening the sensor would
|
||||
throw away the state the pre-roll just rebuilt. Only the capture is
|
||||
replaced, and the tracker's reference frame is preserved.
|
||||
"""
|
||||
sensor = self._sensor
|
||||
open_source = getattr(sensor, "_open_video_source", None)
|
||||
if not callable(open_source):
|
||||
return False
|
||||
initial = getattr(sensor, "initial_img", None)
|
||||
try:
|
||||
import orisys
|
||||
|
||||
sensor.vid_src = orisys.capture.normalize_source(str(segment))
|
||||
except Exception: # noqa: BLE001 - fall back to assigning the path
|
||||
sensor.vid_src = str(segment)
|
||||
sensor._is_live_camera = False
|
||||
open_source(False, None)
|
||||
if initial is not None:
|
||||
# The tracker's reference frame belongs to the session, not to the
|
||||
# file: replacing it would silently reset what "no contact" means.
|
||||
sensor.initial_img = initial
|
||||
return True
|
||||
|
||||
def read(self) -> Optional[np.ndarray]:
|
||||
from orisys_app_shared.acquisition.sdk_worker import process_chain
|
||||
|
||||
image = self._sensor.get_img()
|
||||
if image is None:
|
||||
return None
|
||||
# The same function the acquisition loop runs.
|
||||
depth, error = process_chain(self._sensor)
|
||||
if error:
|
||||
logger.warning("Replay processing chain reported: %s", error)
|
||||
return depth
|
||||
|
||||
def reset(self) -> bool:
|
||||
"""Clear the tracker, exactly as the operator's zeroing did live."""
|
||||
reset = getattr(self._sensor, "reset", None)
|
||||
if not callable(reset):
|
||||
return False
|
||||
reset()
|
||||
return True
|
||||
|
||||
def close(self) -> None:
|
||||
self._sensor.disconnect()
|
||||
|
||||
|
||||
class FingerReplay:
|
||||
"""The finger's live chain: unroll the source frame, then process the tactile."""
|
||||
|
||||
def __init__(self, segment: Path, config_ref: str, export_dir) -> None:
|
||||
from orisys.finger import FingerRuntime
|
||||
|
||||
from orisys_app_shared.acquisition.constants import (
|
||||
FORCE_SAMPLE_RATE_HZ,
|
||||
LOWPASS_CUTOFF_HZ,
|
||||
)
|
||||
|
||||
self._config_ref = config_ref
|
||||
self._export_dir = export_dir
|
||||
kwargs: dict = {
|
||||
"config": config_ref,
|
||||
"video": str(segment),
|
||||
"verbose": False,
|
||||
}
|
||||
if export_dir is not None:
|
||||
kwargs["export_dir"] = str(export_dir)
|
||||
self._runtime = FingerRuntime.open(**kwargs)
|
||||
self._runtime.warmup(color=False)
|
||||
tactile = getattr(self._runtime, "tactile", None)
|
||||
if tactile is not None and hasattr(tactile, "configure"):
|
||||
try:
|
||||
# The same filter settings the acquisition loop applies.
|
||||
tactile.configure(
|
||||
lowpass_cutoff_hz=LOWPASS_CUTOFF_HZ,
|
||||
lowpass_sample_rate_hz=FORCE_SAMPLE_RATE_HZ,
|
||||
)
|
||||
except Exception: # noqa: BLE001 - the defaults still work
|
||||
logger.exception("Could not configure the replayed tactile sensor")
|
||||
|
||||
def set_source(self, segment: Path) -> bool:
|
||||
"""Point the same runtime at another video, keeping the tactile pipeline."""
|
||||
try:
|
||||
from orisys.capture import (
|
||||
CaptureConfig,
|
||||
ReconnectPolicy,
|
||||
is_live_camera,
|
||||
normalize_source,
|
||||
open_capture,
|
||||
)
|
||||
|
||||
runtime = self._runtime
|
||||
video = normalize_source(str(segment))
|
||||
camera = dict(getattr(runtime.geometry, "config", {}).get("camera", {}))
|
||||
capture_config = CaptureConfig.from_camera_dict(
|
||||
video,
|
||||
camera,
|
||||
reconnect=ReconnectPolicy(enabled=False),
|
||||
verbose=False,
|
||||
background=False,
|
||||
)
|
||||
capture_source = open_capture(capture_config, start_background=False)
|
||||
if not capture_source.is_open():
|
||||
return False
|
||||
if is_live_camera(video):
|
||||
capture_source.start_background()
|
||||
previous = runtime.capture_source
|
||||
runtime.capture_source = capture_source
|
||||
runtime.capture = capture_source.cap
|
||||
runtime.capture_worker = capture_source
|
||||
if previous is not None:
|
||||
try:
|
||||
previous.close()
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("Could not close the previous finger source")
|
||||
return True
|
||||
except Exception: # noqa: BLE001 - reported as an approximate replay
|
||||
logger.exception("Could not swap the finger video source")
|
||||
return False
|
||||
|
||||
def read(self) -> Optional[np.ndarray]:
|
||||
from orisys_app_shared.acquisition.sdk_worker import process_chain
|
||||
|
||||
ok, source_frame = self._runtime.read_source()
|
||||
if not ok or source_frame is None:
|
||||
return None
|
||||
unrolled = self._runtime.process_frame(source_frame, color=False)
|
||||
if unrolled is None:
|
||||
return None
|
||||
depth, error = process_chain(self._runtime.tactile)
|
||||
if error:
|
||||
logger.warning("Replay processing chain reported: %s", error)
|
||||
return depth
|
||||
|
||||
def reset(self) -> bool:
|
||||
"""Clear the tactile tracker, as the operator's zeroing did live."""
|
||||
tactile = getattr(self._runtime, "tactile", None)
|
||||
target = tactile if callable(getattr(tactile, "reset", None)) else None
|
||||
if target is None:
|
||||
sensor = getattr(tactile, "raw_sensor", None)
|
||||
target = sensor if callable(getattr(sensor, "reset", None)) else None
|
||||
if target is None:
|
||||
return False
|
||||
target.reset()
|
||||
return True
|
||||
|
||||
def close(self) -> None:
|
||||
self._runtime.close()
|
||||
|
||||
|
||||
def open_replay(sensor_type: str, segment: Path, config_ref: str, export_dir=None):
|
||||
"""The backend for a recorded sensor type."""
|
||||
if str(sensor_type) == "finger":
|
||||
return FingerReplay(segment, config_ref, export_dir)
|
||||
return BoxReplay(segment, config_ref)
|
||||
@@ -41,6 +41,7 @@ import cv2
|
||||
import numpy as np
|
||||
|
||||
from orisys_app_shared.record.sample import RecordSample, to_record_sample
|
||||
from orisys_app_shared.record.video_writer import VideoSegments
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -87,7 +88,8 @@ MAX_CONSECUTIVE_ERRORS = 5
|
||||
ALARM_EVERY_N_DROPS = 1000
|
||||
#: How many lost-row gaps may be remembered for the frame index.
|
||||
GAP_BACKLOG = 4096
|
||||
#: How many data rows may be parked while the item queue is full of video.
|
||||
#: Hard cap on the ordered stream. Past this the producer has to give up a row,
|
||||
#: and it says so by writing a gap marker instead.
|
||||
ROW_BACKLOG = 8192
|
||||
|
||||
|
||||
@@ -115,6 +117,9 @@ class RecordConfig:
|
||||
fps: float = 30.0
|
||||
codec: str = "mp4v"
|
||||
segment_seconds: float = 300.0
|
||||
#: Stream depth at which **video** starts being shed. Data rows are not
|
||||
#: dropped by this limit: a row is tiny, and losing one because the encoder
|
||||
#: fell behind is exactly the defect this bound must not reintroduce.
|
||||
max_queue: int = 2048
|
||||
#: Byte budget for the video queue. The item count alone is not a bound: at
|
||||
#: 1920x1080 BGR, 2048 queued frame copies would be ~11.9 GiB.
|
||||
@@ -206,14 +211,16 @@ class ExperimentRecorder:
|
||||
|
||||
def __init__(self, config: Optional[RecordConfig] = None) -> None:
|
||||
self.config = config or RecordConfig()
|
||||
self._queue: queue.Queue = queue.Queue(maxsize=max(16, self.config.max_queue))
|
||||
#: Bounded side channel for frames whose data row was lost to a full
|
||||
#: queue. It is tiny, so it never grows with the recording.
|
||||
#: One ordered stream for rows and events. It is **unbounded** on count:
|
||||
#: the producer checks ``ROW_BACKLOG``/``max_queue`` itself, so a slow
|
||||
#: encoder can no longer make the *queue* quietly delete a data row, and
|
||||
#: rows are written in the order they were acquired.
|
||||
self._queue: queue.Queue = queue.Queue()
|
||||
#: Stream positions of frames whose data row was lost. Kept out of the
|
||||
#: stream because there is no room for them there; flushed in order.
|
||||
self._gaps: deque = deque(maxlen=GAP_BACKLOG)
|
||||
#: Rows that did not fit in the item queue because it was full of video.
|
||||
#: They are tiny, so a small backlog covers any realistic stall.
|
||||
self._row_backlog: deque = deque(maxlen=ROW_BACKLOG)
|
||||
self._queued_bytes = 0
|
||||
self._seq = 0
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
self._stop = threading.Event()
|
||||
self._lock = threading.Lock()
|
||||
@@ -242,6 +249,8 @@ class ExperimentRecorder:
|
||||
self._preroll_segment: Optional[str] = None
|
||||
#: Problems the writer found while closing, carried into the report.
|
||||
self._writer_problems: list[str] = []
|
||||
#: Coefficient files stored inside the recording directory.
|
||||
self._calibration_snapshots: list[dict] = []
|
||||
|
||||
# ------------------------------------------------------------------ state
|
||||
@property
|
||||
@@ -318,7 +327,20 @@ class ExperimentRecorder:
|
||||
self._state = STATE_STARTING
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
directory = Path(root) / f"{stamp}_session{session_id:04d}"
|
||||
try:
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
except Exception as exc: # noqa: BLE001 - reported to the caller
|
||||
# A directory that cannot be created is a *terminal* failure, not a
|
||||
# stuck ``starting``: the operator can fix the path and press record
|
||||
# again, and the recorder must be willing to try.
|
||||
with self._lock:
|
||||
self._stats.failed = True
|
||||
self._stats.last_error = f"could not create {directory}: {exc}"
|
||||
self._state = STATE_FAILED
|
||||
self._accepting = False
|
||||
self._directory = None
|
||||
self._alarm(f"could not create the recording directory: {exc}")
|
||||
raise
|
||||
with self._lock:
|
||||
self._directory = directory
|
||||
self._stats = RecordStats()
|
||||
@@ -335,9 +357,10 @@ class ExperimentRecorder:
|
||||
self._preroll_frames = 0
|
||||
self._preroll_segment = None
|
||||
self._writer_problems = []
|
||||
self._calibration_snapshots = []
|
||||
# Stale bookkeeping from a previous run must not leak into this one.
|
||||
self._gaps.clear()
|
||||
self._row_backlog.clear()
|
||||
self._seq = 0
|
||||
self._queued_bytes = 0
|
||||
# Accept immediately: see ``_accepts_packets``.
|
||||
self._accepting = True
|
||||
@@ -370,26 +393,26 @@ class ExperimentRecorder:
|
||||
self._state = STATE_STOPPING
|
||||
self._accepting = False
|
||||
self._stop.set()
|
||||
try:
|
||||
self._queue.put_nowait(("stop", None))
|
||||
except queue.Full: # pragma: no cover - writer will drain and see the flag
|
||||
pass
|
||||
# The stop marker is an ordinary item in the stream: everything acquired
|
||||
# before it is written before it, and the writer stops after it.
|
||||
self._enqueue(("stop", self._next_seq(), None))
|
||||
thread = self._thread
|
||||
if wait and thread is not None and thread.is_alive():
|
||||
thread.join(timeout=timeout)
|
||||
if thread is not None and thread.is_alive():
|
||||
# Still writing: do not claim the experiment was recorded.
|
||||
report = RecordReport(
|
||||
# Either we were asked not to wait, or the writer missed the deadline.
|
||||
# A finished recording is only ever reported by the thread that closed
|
||||
# the files, so this is explicitly *not* a completion — and it is not
|
||||
# cached as the report either, so a later stop() still sees the real
|
||||
# one (``recorder.report`` stays None until then).
|
||||
return RecordReport(
|
||||
directory=self._directory,
|
||||
stats=self.stats(),
|
||||
problems=[
|
||||
"the recording writer did not stop within "
|
||||
f"{timeout:g}s; the files are not closed yet"
|
||||
"the recording writer has not finished closing the files yet"
|
||||
],
|
||||
completed=False,
|
||||
)
|
||||
self._report = report
|
||||
return report
|
||||
self._thread = None
|
||||
return self._report or self._finalise()
|
||||
|
||||
@@ -398,13 +421,34 @@ class ExperimentRecorder:
|
||||
|
||||
A session can outlive the coefficients it started with, so the version is
|
||||
per packet, not per session: the acquisition thread stamps every packet
|
||||
with the calibration that was loaded when it was acquired.
|
||||
with the calibration that was loaded when it was acquired. An empty
|
||||
version is meaningful — it says the device has no calibration loaded.
|
||||
"""
|
||||
with self._lock:
|
||||
self._calibration_version = str(version or "")
|
||||
if self.is_recording:
|
||||
self.log_event("calibration_version", self._calibration_version)
|
||||
|
||||
def note_calibration_snapshot(
|
||||
self,
|
||||
*,
|
||||
version: str,
|
||||
name: str,
|
||||
frame_id: int = -1,
|
||||
mono_t: Optional[float] = None,
|
||||
) -> None:
|
||||
"""Remember a coefficient file stored inside the recording directory."""
|
||||
entry = {
|
||||
"version": str(version or ""),
|
||||
"file": str(name or ""),
|
||||
"frame_id": int(frame_id),
|
||||
"mono_t": None if mono_t is None else round(float(mono_t), 6),
|
||||
"at": datetime.now(timezone.utc).isoformat(timespec="milliseconds"),
|
||||
}
|
||||
with self._lock:
|
||||
self._calibration_snapshots.append(entry)
|
||||
logger.info("Calibration snapshot stored: %s", name)
|
||||
|
||||
def log_event(
|
||||
self,
|
||||
kind: str,
|
||||
@@ -433,21 +477,29 @@ class ExperimentRecorder:
|
||||
"kind": str(kind),
|
||||
"detail": str(detail or ""),
|
||||
}
|
||||
try:
|
||||
self._queue.put_nowait(("event", event))
|
||||
except queue.Full:
|
||||
# Events ride the same ordered stream as the rows, so an operator action
|
||||
# cannot overtake the samples it was taken between.
|
||||
if self._queue.qsize() >= ROW_BACKLOG:
|
||||
# The stream is saturated: an event is the cheapest thing to give up,
|
||||
# but the loss is counted rather than silent.
|
||||
self._note_drop("event")
|
||||
return
|
||||
self._enqueue(("event", self._next_seq(), event))
|
||||
|
||||
# ----------------------------------------------------------- sink interface
|
||||
def on_packet(self, packet) -> None:
|
||||
"""Called on the acquisition thread: enqueue, never do I/O here.
|
||||
|
||||
A data row is small and a frame is large, so video is what gives way.
|
||||
When the byte budget is exhausted the **frame** is shed, and when even
|
||||
the item queue is full the frame is shed too and the row is parked in a
|
||||
small backlog. Losing the row because the video could not keep up is what
|
||||
made a video overflow look like a video-only loss while CSV rows quietly
|
||||
disappeared.
|
||||
A data row is small and a frame is large, so video is what gives way —
|
||||
and the two are decided **separately**:
|
||||
|
||||
* the rows go into one ordered, unbounded stream (bounded only by
|
||||
``ROW_BACKLOG``, at which point the loss is counted and marked);
|
||||
* the frames are bounded by a byte budget, and a frame that would exceed
|
||||
it is shed *before* it is queued, so nothing has to be rolled back.
|
||||
|
||||
The stream being ordered is what makes the files ordered: rows are written
|
||||
in the order they were acquired, no matter how far behind the encoder is.
|
||||
"""
|
||||
if not self._accepts_packets():
|
||||
with self._lock:
|
||||
@@ -462,36 +514,53 @@ class ExperimentRecorder:
|
||||
with self._lock:
|
||||
self._last_t = sample.mono_t
|
||||
|
||||
# Depth is only consulted to decide the *video*: a row is never refused
|
||||
# because the encoder is behind, only when the stream itself is full.
|
||||
depth = self._queue.qsize()
|
||||
if depth >= ROW_BACKLOG:
|
||||
self._note_gap(sample)
|
||||
return
|
||||
|
||||
frame = None
|
||||
if self.config.video or self.config.lossless_frames:
|
||||
candidate = self._capture_frame(packet)
|
||||
size = 0 if candidate is None else int(candidate.nbytes)
|
||||
with self._lock:
|
||||
affordable = (self._queued_bytes + size) <= max(
|
||||
1024, int(self.config.max_queue_bytes)
|
||||
)
|
||||
if candidate is None:
|
||||
pass
|
||||
elif affordable:
|
||||
if candidate is not None:
|
||||
size = int(candidate.nbytes)
|
||||
if depth >= max(16, int(self.config.max_queue)):
|
||||
self._shed_video("record queue overflow")
|
||||
elif not self._affordable(size):
|
||||
self._shed_video("video byte budget reached")
|
||||
else:
|
||||
frame = candidate
|
||||
# Reserved here and released exactly once, by the writer that
|
||||
# consumes it (see ``_handle_frame``).
|
||||
with self._lock:
|
||||
self._queued_bytes += size
|
||||
else:
|
||||
self._shed_video("video byte budget reached")
|
||||
try:
|
||||
self._queue.put_nowait(("frame", (sample, frame)))
|
||||
except queue.Full:
|
||||
if frame is not None:
|
||||
# The queue is full of video: the row is tiny and must survive.
|
||||
frame = None
|
||||
self._shed_video("record queue overflow")
|
||||
self._enqueue(("frame", self._next_seq(), sample, frame))
|
||||
|
||||
def _next_seq(self) -> int:
|
||||
with self._lock:
|
||||
room = len(self._row_backlog) < self._row_backlog.maxlen
|
||||
if room:
|
||||
self._row_backlog.append(sample)
|
||||
if not room:
|
||||
# Even the row backlog is full: the hole is at least marked.
|
||||
self._note_gap(sample)
|
||||
self._seq += 1
|
||||
return self._seq
|
||||
|
||||
def _affordable(self, size: int) -> bool:
|
||||
with self._lock:
|
||||
return (self._queued_bytes + int(size)) <= max(
|
||||
1024, int(self.config.max_queue_bytes)
|
||||
)
|
||||
|
||||
def _enqueue(self, item) -> bool:
|
||||
"""Append to the ordered stream. False when the stream is full.
|
||||
|
||||
The queue itself is unbounded — its depth is bounded by ``ROW_BACKLOG``
|
||||
and checked by the callers — so a slow writer can never make the producer
|
||||
drop data behind its back.
|
||||
"""
|
||||
try:
|
||||
self._queue.put_nowait(item)
|
||||
return True
|
||||
except queue.Full: # pragma: no cover - the queue is unbounded
|
||||
return False
|
||||
|
||||
def _shed_video(self, reason: str) -> None:
|
||||
with self._lock:
|
||||
@@ -505,14 +574,26 @@ class ExperimentRecorder:
|
||||
)
|
||||
|
||||
def _note_gap(self, sample) -> None:
|
||||
"""Remember a frame whose row was lost, so the replay can show the hole."""
|
||||
"""Remember a frame whose row was lost, so the replay can show the hole.
|
||||
|
||||
The gap carries the stream position it happened at. That is what lets the
|
||||
writer put the hole in the *right place* in ``frames.csv`` instead of
|
||||
appending it after newer frames (which would make the index unsorted).
|
||||
"""
|
||||
with self._lock:
|
||||
self._stats.dropped_data += 1
|
||||
if len(self._gaps) >= self._gaps.maxlen:
|
||||
# The gap backlog is bounded, so a long overflow loses the marker.
|
||||
self._stats.unrecorded_gaps += 1
|
||||
return
|
||||
self._gaps.append((sample.frame_id, sample.mono_t, sample.wall_t))
|
||||
self._gaps.append(
|
||||
(
|
||||
self._seq + 1,
|
||||
sample.frame_id,
|
||||
sample.mono_t,
|
||||
sample.wall_t,
|
||||
)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _capture_frame(packet):
|
||||
@@ -574,6 +655,7 @@ class ExperimentRecorder:
|
||||
session_id=self._session_id,
|
||||
calibration_id=self._calibration_id,
|
||||
preroll=self._preroll,
|
||||
on_problem=self._record_failure,
|
||||
)
|
||||
# Written once the pre-roll file exists, so session.json never names
|
||||
# a file that was not created.
|
||||
@@ -583,26 +665,40 @@ class ExperimentRecorder:
|
||||
self._preroll = []
|
||||
self._write_session_json(final=False)
|
||||
with self._lock:
|
||||
# A stop *or a failure* that arrived while the files were opening
|
||||
# must not be undone here: accepting packets again after ``stop``
|
||||
# would make a finished recording look alive, and overwriting
|
||||
# ``failed`` with ``recording`` would hide a dead writer. The
|
||||
# stream is still drained — those rows were accepted.
|
||||
if self._stop.is_set() or self._state in (
|
||||
STATE_STOPPING,
|
||||
STATE_FAILED,
|
||||
):
|
||||
self._accepting = False
|
||||
else:
|
||||
self._state = STATE_RECORDING
|
||||
self._accepting = True
|
||||
while True:
|
||||
# Parked rows and gaps first: a stop request must not strand the
|
||||
# data this session already accepted.
|
||||
self._drain_rows(writer)
|
||||
self._drain_gaps(writer)
|
||||
# Gaps first, but only those that happened *before* the item about
|
||||
# to be written: the hole must appear in the frame index where it
|
||||
# belongs, not after newer frames.
|
||||
if self._stop.is_set() and self._queue.empty():
|
||||
self._flush_gaps(writer, None)
|
||||
break
|
||||
try:
|
||||
kind, payload = self._queue.get(timeout=0.2)
|
||||
item = self._queue.get(timeout=0.2)
|
||||
except queue.Empty:
|
||||
writer.maybe_flush()
|
||||
continue
|
||||
kind = item[0]
|
||||
seq = item[1]
|
||||
self._flush_gaps(writer, seq)
|
||||
if kind == "stop":
|
||||
break
|
||||
if kind == "event":
|
||||
self._handle_event(writer, payload)
|
||||
self._handle_event(writer, item[2])
|
||||
continue
|
||||
sample, frame = payload
|
||||
_kind, _seq, sample, frame = item
|
||||
self._handle_frame(writer, sample, frame)
|
||||
writer.maybe_flush()
|
||||
except Exception as exc: # noqa: BLE001 - reported through stats
|
||||
@@ -617,32 +713,31 @@ class ExperimentRecorder:
|
||||
self._accepting = False
|
||||
problems: list[str] = []
|
||||
if writer is not None:
|
||||
# A packet accepted just before the stop flag may have left its
|
||||
# row or gap behind; write those before the file is closed.
|
||||
self._drain_rows(writer)
|
||||
self._drain_gaps(writer)
|
||||
# A row or gap accepted just before the stop flag may still be in
|
||||
# the stream; write those before the file is closed.
|
||||
self._flush_gaps(writer, None)
|
||||
try:
|
||||
problems = writer.close()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
problems = [f"close failed: {type(exc).__name__}: {exc}"]
|
||||
with self._lock:
|
||||
self._stats.write_errors += writer.write_errors
|
||||
self._stats.segments = writer.segments
|
||||
self._stats.video_frames = writer.video_frames
|
||||
self._stats.bytes_written = writer.bytes_written
|
||||
# Closing problems are failures of the same recording and are counted
|
||||
# exactly once, through the same path as a failed row, flush or event.
|
||||
for problem in problems:
|
||||
self._record_failure(problem)
|
||||
self._alarm(problem)
|
||||
with self._lock:
|
||||
if problems or failure:
|
||||
self._stats.failed = True
|
||||
if not self._stats.last_error:
|
||||
self._stats.last_error = problems[0] if problems else failure
|
||||
if failure and not self._stats.last_error:
|
||||
self._stats.last_error = failure
|
||||
# The writer's own findings (a flush that failed, a pre-roll that
|
||||
# could not be stored) belong in the report: an operator reading
|
||||
# only the terminal report must not have to grep the log.
|
||||
self._writer_problems = list(problems)
|
||||
failed = self._stats.failed
|
||||
self._state = STATE_FAILED if failed else STATE_STOPPED
|
||||
for problem in problems:
|
||||
self._alarm(problem)
|
||||
# Only now, with the files closed, is the report trustworthy.
|
||||
self._report = self._finalise(completed=True)
|
||||
|
||||
@@ -657,6 +752,10 @@ class ExperimentRecorder:
|
||||
def _handle_frame(self, writer, sample, frame) -> None:
|
||||
try:
|
||||
if frame is not None:
|
||||
# The one release for the reservation taken in ``on_packet``.
|
||||
# A frame is enqueued once and consumed once, so this cannot
|
||||
# double-count; doing it *before* the write also means a failed
|
||||
# write does not leak the budget for a frame that is gone.
|
||||
with self._lock:
|
||||
self._queued_bytes = max(0, self._queued_bytes - int(frame.nbytes))
|
||||
writer.write_sample(sample, frame)
|
||||
@@ -668,22 +767,21 @@ class ExperimentRecorder:
|
||||
except Exception as exc: # noqa: BLE001
|
||||
self._record_error(exc)
|
||||
|
||||
def _drain_rows(self, writer) -> None:
|
||||
"""Write rows parked while the item queue was full of video."""
|
||||
while True:
|
||||
with self._lock:
|
||||
if not self._row_backlog:
|
||||
return
|
||||
sample = self._row_backlog.popleft()
|
||||
self._handle_frame(writer, sample, None)
|
||||
def _flush_gaps(self, writer, upto_seq) -> None:
|
||||
"""Write the frame-index rows for frames whose data row was lost.
|
||||
|
||||
def _drain_gaps(self, writer) -> None:
|
||||
"""Write the frame-index rows for frames whose data row was lost."""
|
||||
Gaps are written in stream order and only up to ``upto_seq`` (``None``
|
||||
flushes the rest), so ``frames.csv`` stays sorted by time even when the
|
||||
loss happened while the writer was far behind.
|
||||
"""
|
||||
while True:
|
||||
with self._lock:
|
||||
if not self._gaps:
|
||||
return
|
||||
frame_id, mono_t, wall_t = self._gaps.popleft()
|
||||
seq, frame_id, mono_t, wall_t = self._gaps[0]
|
||||
if upto_seq is not None and seq > upto_seq:
|
||||
return
|
||||
self._gaps.popleft()
|
||||
try:
|
||||
writer.write_gap(frame_id, mono_t, wall_t)
|
||||
with self._lock:
|
||||
@@ -693,18 +791,32 @@ class ExperimentRecorder:
|
||||
return
|
||||
|
||||
def _record_error(self, exc: Exception) -> None:
|
||||
self._record_failure(f"{type(exc).__name__}: {exc}")
|
||||
|
||||
def _record_failure(self, message: str, *, fatal: bool = False) -> None:
|
||||
"""One failed write, flush, close or metadata update.
|
||||
|
||||
Every failure enters the state machine here, so the counters, the fatal
|
||||
transition and the report cannot disagree about what happened. A writer
|
||||
that kept its own count (a failed ``maybe_flush`` used to) left the
|
||||
recording looking healthy until the very end.
|
||||
|
||||
``fatal`` marks a failure that ends the recording on its own: a recording
|
||||
whose ``session.json`` could not be written is not a recording anyone can
|
||||
identify afterwards.
|
||||
"""
|
||||
with self._lock:
|
||||
self._stats.write_errors += 1
|
||||
self._stats.last_error = f"{type(exc).__name__}: {exc}"
|
||||
self._stats.last_error = str(message)
|
||||
self._consecutive_errors += 1
|
||||
fatal = self._consecutive_errors >= MAX_CONSECUTIVE_ERRORS
|
||||
fatal = fatal or self._consecutive_errors >= MAX_CONSECUTIVE_ERRORS
|
||||
if fatal:
|
||||
self._stats.failed = True
|
||||
# Stop accepting immediately: consuming packets that cannot be
|
||||
# written only hides the loss behind a "recording" state.
|
||||
self._accepting = False
|
||||
self._state = STATE_FAILED
|
||||
self._alarm(f"recording write failed: {exc}")
|
||||
self._alarm(f"recording write failed: {message}")
|
||||
if fatal:
|
||||
self._stop.set()
|
||||
|
||||
@@ -712,6 +824,10 @@ class ExperimentRecorder:
|
||||
with self._lock:
|
||||
if self._t0 is not None and self._last_t is not None:
|
||||
self._stats.duration_s = round(float(self._last_t - self._t0), 6)
|
||||
# Metadata **before** the report is assembled: a session.json that could
|
||||
# not be written is part of what happened, and used to be discovered only
|
||||
# after the report had already been built from the older statistics.
|
||||
self._write_session_json(final=completed)
|
||||
stats = self.stats()
|
||||
problems: list[str] = []
|
||||
if stats.dropped_data or stats.dropped_video:
|
||||
@@ -734,8 +850,6 @@ class ExperimentRecorder:
|
||||
for problem in self._writer_problems:
|
||||
if problem not in problems:
|
||||
problems.append(problem)
|
||||
# ``ended_at`` is only written once the files are actually closed.
|
||||
self._write_session_json(final=completed)
|
||||
return RecordReport(
|
||||
directory=self._directory,
|
||||
stats=stats,
|
||||
@@ -763,10 +877,11 @@ class ExperimentRecorder:
|
||||
)
|
||||
temporary.replace(target)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
with self._lock:
|
||||
self._stats.write_errors += 1
|
||||
self._stats.last_error = f"session.json: {type(exc).__name__}: {exc}"
|
||||
self._alarm(f"could not write session.json: {exc}")
|
||||
# The same single failure path as a write: it is counted, it ends the
|
||||
# recording, and it reaches the report.
|
||||
self._record_failure(
|
||||
f"session.json: {type(exc).__name__}: {exc}", fatal=True
|
||||
)
|
||||
|
||||
def _build_metadata(self, *, final: bool) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
@@ -783,6 +898,9 @@ class ExperimentRecorder:
|
||||
"video": self.config.as_dict(),
|
||||
"events": events,
|
||||
"stats": stats,
|
||||
# The coefficients the session actually ran with: the file itself is
|
||||
# copied into the recording, so a re-export cannot rewrite history.
|
||||
"calibration_snapshots": [dict(entry) for entry in self._calibration_snapshots],
|
||||
# The replay needs these to know whether the algorithm state at the
|
||||
# start of the recording can be rebuilt at all.
|
||||
"preroll_frames": int(self._preroll_frames),
|
||||
@@ -802,16 +920,36 @@ class ExperimentRecorder:
|
||||
class _SessionWriter:
|
||||
"""Owns the file handles. Only ever touched by the writer thread."""
|
||||
|
||||
def _note_problem(self, message: str) -> None:
|
||||
"""Record a failure here *and* hand it to the recorder at once."""
|
||||
self.write_errors += 1
|
||||
if len(self._problems) < 5:
|
||||
self._problems.append(message)
|
||||
if self._on_problem is not None:
|
||||
try:
|
||||
self._on_problem(message)
|
||||
except Exception: # noqa: BLE001 - reporting must not break writing
|
||||
logger.exception("Recording problem handler failed")
|
||||
|
||||
def __init__(
|
||||
self, *, directory, config, session_id, calibration_id, preroll=()
|
||||
self,
|
||||
*,
|
||||
directory,
|
||||
config,
|
||||
session_id,
|
||||
calibration_id,
|
||||
preroll=(),
|
||||
on_problem=None,
|
||||
) -> None:
|
||||
self.directory = Path(directory)
|
||||
self.config = config
|
||||
self.session_id = session_id
|
||||
self.calibration_id = calibration_id
|
||||
#: ``on_problem(message)`` reports a failure to the recorder's state
|
||||
#: machine *immediately*. A private counter here was invisible until the
|
||||
#: files were closed, so a failing flush looked like a healthy recording.
|
||||
self._on_problem = on_problem
|
||||
self.write_errors = 0
|
||||
self.segments = 0
|
||||
self.video_frames = 0
|
||||
self.png_frames = 0
|
||||
self.gap_rows = 0
|
||||
self.bytes_written = 0
|
||||
@@ -820,12 +958,14 @@ class _SessionWriter:
|
||||
self.preroll_segment: Optional[str] = None
|
||||
self._last_flush = time.perf_counter()
|
||||
self._t0: Optional[float] = None
|
||||
self._segment: Optional[cv2.VideoWriter] = None
|
||||
self._segment_index = 0
|
||||
self._segment_frames = 0
|
||||
self._segment_started = 0.0
|
||||
self._frame_shape: Optional[tuple[int, int]] = None
|
||||
self._problems: list[str] = []
|
||||
#: The video side, in its own component: one rolling segment at a time.
|
||||
self._video = VideoSegments(
|
||||
self.directory,
|
||||
codec=self.config.codec,
|
||||
fps=self.config.fps,
|
||||
segment_seconds=self.config.segment_seconds,
|
||||
)
|
||||
|
||||
self._forces = self._open_csv("forces.csv", FORCES_HEADER)
|
||||
self._frames = self._open_csv("frames.csv", FRAMES_HEADER)
|
||||
@@ -863,22 +1003,8 @@ class _SessionWriter:
|
||||
kept.append(frame)
|
||||
used += size
|
||||
kept.reverse()
|
||||
self._frame_shape = (int(kept[0].shape[1]), int(kept[0].shape[0]))
|
||||
name = "preroll_0001.mp4"
|
||||
path = self.directory / name
|
||||
fourcc = cv2.VideoWriter_fourcc(*self.config.codec)
|
||||
writer = cv2.VideoWriter(
|
||||
str(path), fourcc, max(1.0, float(self.config.fps)), self._frame_shape
|
||||
)
|
||||
if not writer.isOpened():
|
||||
raise OSError(f"could not open video writer for {name}")
|
||||
try:
|
||||
for frame in kept:
|
||||
writer.write(frame)
|
||||
self.preroll_frames += 1
|
||||
finally:
|
||||
writer.release()
|
||||
self.preroll_segment = name if self.preroll_frames else None
|
||||
self.preroll_frames = self._video.write_sequence(kept)
|
||||
self.preroll_segment = "preroll_0001.mp4" if self.preroll_frames else None
|
||||
|
||||
def _open_csv(self, name: str, header):
|
||||
handle = (self.directory / name).open("w", newline="", encoding="utf-8")
|
||||
@@ -957,16 +1083,15 @@ class _SessionWriter:
|
||||
self.gap_rows += 1
|
||||
|
||||
def _write_video_frame(self, frame) -> tuple[int, int]:
|
||||
if self._frame_shape is None:
|
||||
self._frame_shape = (int(frame.shape[1]), int(frame.shape[0]))
|
||||
if self._should_roll():
|
||||
self._close_segment()
|
||||
if self._segment is None:
|
||||
self._open_segment()
|
||||
self._segment.write(frame)
|
||||
self._segment_frames += 1
|
||||
self.video_frames += 1
|
||||
return (self._segment_index, self._segment_frames)
|
||||
return self._video.write(frame)
|
||||
|
||||
@property
|
||||
def video_frames(self) -> int:
|
||||
return self._video.video_frames
|
||||
|
||||
@property
|
||||
def segments(self) -> int:
|
||||
return self._video.segments
|
||||
|
||||
def _write_lossless(self, frame) -> None:
|
||||
self.png_frames += 1
|
||||
@@ -989,33 +1114,6 @@ class _SessionWriter:
|
||||
)
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------ segments
|
||||
def _should_roll(self) -> bool:
|
||||
if self._segment is None:
|
||||
return False
|
||||
if self.config.segment_seconds <= 0:
|
||||
return False
|
||||
return (time.perf_counter() - self._segment_started) >= self.config.segment_seconds
|
||||
|
||||
def _open_segment(self) -> None:
|
||||
self._segment_index += 1
|
||||
path = self.directory / f"raw_{self._segment_index:04d}.mp4"
|
||||
fourcc = cv2.VideoWriter_fourcc(*self.config.codec)
|
||||
fps = max(1.0, float(self.config.fps))
|
||||
writer = cv2.VideoWriter(str(path), fourcc, fps, self._frame_shape)
|
||||
if not writer.isOpened():
|
||||
raise OSError(f"could not open video writer for {path.name}")
|
||||
self._segment = writer
|
||||
self._segment_frames = 0
|
||||
self._segment_started = time.perf_counter()
|
||||
self.segments = self._segment_index
|
||||
|
||||
def _close_segment(self) -> None:
|
||||
if self._segment is None:
|
||||
return
|
||||
self._segment.release()
|
||||
self._segment = None
|
||||
|
||||
def maybe_flush(self) -> None:
|
||||
if (time.perf_counter() - self._last_flush) < self.config.flush_interval_s:
|
||||
return
|
||||
@@ -1024,23 +1122,22 @@ class _SessionWriter:
|
||||
try:
|
||||
handle.flush()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
self.write_errors += 1
|
||||
if len(self._problems) < 5:
|
||||
self._problems.append(f"flush failed: {exc}")
|
||||
self._note_problem(f"flush failed: {exc}")
|
||||
|
||||
def close(self) -> list[str]:
|
||||
self._close_segment()
|
||||
try:
|
||||
self._video.close()
|
||||
except Exception as exc: # noqa: BLE001 - the CSVs still have to close
|
||||
self._note_problem(f"video close failed: {exc}")
|
||||
for handle, _writer in (self._forces, self._frames, self._events):
|
||||
try:
|
||||
handle.flush()
|
||||
handle.close()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
self.write_errors += 1
|
||||
if len(self._problems) < 5:
|
||||
self._problems.append(f"close failed: {exc}")
|
||||
self._note_problem(f"close failed: {exc}")
|
||||
try:
|
||||
self.bytes_written = sum(
|
||||
path.stat().st_size for path in self.directory.glob("*") if path.is_file()
|
||||
path.stat().st_size for path in self.directory.rglob("*") if path.is_file()
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
self.bytes_written = 0
|
||||
|
||||
@@ -0,0 +1,453 @@
|
||||
"""A time index over a recorded ``forces.csv``, so history is read on demand.
|
||||
|
||||
The replay used to load every row of a recording into Python lists and then into
|
||||
arrays. That is fine for a minute and hopeless for the multi-hour recordings the
|
||||
store is designed for, and it made the charts the only place where "the whole
|
||||
session" had to exist in memory at once.
|
||||
|
||||
This index keeps the *small* things in memory — one time per row, one validity
|
||||
flag, one frame id, plus periodic byte offsets — and reads the *values* only for
|
||||
the interval being drawn. It also builds a multi-level overview (per-bucket
|
||||
min/max/count) while streaming the file once, which is what a whole-session strip
|
||||
needs and what it cannot get by sampling a few thousand rows.
|
||||
|
||||
Nothing here touches Qt, and nothing here trusts the file to be in order: a
|
||||
legacy recording is sorted at load time (see ``playback.RecordingReader``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import logging
|
||||
from bisect import bisect_left, bisect_right
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Iterator, Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: How many rows to read between two recorded byte offsets. Larger means a
|
||||
#: cheaper interval read but a longer scan to reach the interval.
|
||||
CHECKPOINT_ROWS = 4096
|
||||
#: Default number of overview buckets for the whole-session strip.
|
||||
DEFAULT_BUCKETS = 1200
|
||||
|
||||
_FORCE_COLUMNS = ("FN_raw", "FX_raw", "FY_raw", "FN_N", "FX_N", "FY_N")
|
||||
|
||||
|
||||
def _parse(text: str) -> float:
|
||||
text = (text or "").strip()
|
||||
if not text:
|
||||
# An empty cell means "not available"; it must never become 0.
|
||||
return float("nan")
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
return float("nan")
|
||||
|
||||
|
||||
@dataclass
|
||||
class ForceWindow:
|
||||
"""Rows inside one interval — the same shape a session slice has."""
|
||||
|
||||
times: np.ndarray
|
||||
raw: np.ndarray
|
||||
newtons: np.ndarray
|
||||
valid: np.ndarray
|
||||
frame_ids: np.ndarray
|
||||
#: True when the rows were decimated to satisfy the caller's point budget.
|
||||
decimated: bool = False
|
||||
|
||||
def __len__(self) -> int:
|
||||
return int(self.times.size)
|
||||
|
||||
def value_for_unit(self, unit: str) -> np.ndarray:
|
||||
if str(unit).upper().startswith("N"):
|
||||
return self.newtons
|
||||
return self.raw
|
||||
|
||||
def plottable(self, unit: str):
|
||||
"""``(times, values, mask)`` with a per-axis mask — the shared rule."""
|
||||
from orisys_app_shared.record.sample import plottable_axis_mask
|
||||
|
||||
values = self.value_for_unit(unit)
|
||||
return self.times, values, plottable_axis_mask(values, self.valid)
|
||||
|
||||
def plottable_axis(self, unit: str, index: int):
|
||||
from orisys_app_shared.record.sample import plottable_axis_mask
|
||||
|
||||
values = self.value_for_unit(unit)
|
||||
mask = plottable_axis_mask(values, self.valid)[:, index]
|
||||
return self.times[mask], values[mask, index]
|
||||
|
||||
def statistics(self, unit: str) -> "WindowStats":
|
||||
values = self.value_for_unit(unit)
|
||||
mask = np.isfinite(values) & self.valid[:, None]
|
||||
peaks: list[float] = []
|
||||
means: list[float] = []
|
||||
stds: list[float] = []
|
||||
for index in range(3):
|
||||
column = values[:, index][mask[:, index]]
|
||||
if column.size == 0:
|
||||
peaks.append(float("nan"))
|
||||
means.append(float("nan"))
|
||||
stds.append(float("nan"))
|
||||
continue
|
||||
peaks.append(float(np.nanmax(column)))
|
||||
means.append(float(np.nanmean(column)))
|
||||
stds.append(float(np.nanstd(column)))
|
||||
span = 0.0
|
||||
if self.times.size > 1:
|
||||
span = float(self.times[-1] - self.times[0])
|
||||
return WindowStats(
|
||||
duration_s=span,
|
||||
samples=int(mask.any(axis=1).sum()),
|
||||
peaks=tuple(peaks),
|
||||
means=tuple(means),
|
||||
stds=tuple(stds),
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WindowStats:
|
||||
duration_s: float = 0.0
|
||||
samples: int = 0
|
||||
peaks: tuple[float, float, float] = (float("nan"),) * 3
|
||||
means: tuple[float, float, float] = (float("nan"),) * 3
|
||||
stds: tuple[float, float, float] = (float("nan"),) * 3
|
||||
|
||||
|
||||
@dataclass
|
||||
class Overview:
|
||||
"""A whole-session envelope, cheap enough to keep in memory."""
|
||||
|
||||
times: np.ndarray
|
||||
lows: np.ndarray # (buckets, 3)
|
||||
highs: np.ndarray # (buckets, 3)
|
||||
counts: np.ndarray # (buckets,)
|
||||
unit: str = "raw"
|
||||
|
||||
def __len__(self) -> int:
|
||||
return int(self.times.size)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Checkpoint:
|
||||
row: int
|
||||
time: float
|
||||
offset: int
|
||||
|
||||
|
||||
class RecordingIndex:
|
||||
"""Index one recording directory's ``forces.csv`` (and its validity flags)."""
|
||||
|
||||
def __init__(self, directory: str | Path, *, checkpoint_rows: int = CHECKPOINT_ROWS):
|
||||
self.directory = Path(directory)
|
||||
self.checkpoint_rows = max(1, int(checkpoint_rows))
|
||||
self.times = np.empty(0, dtype=np.float64)
|
||||
self.valid = np.empty(0, dtype=bool)
|
||||
self.frame_ids = np.empty(0, dtype=np.int64)
|
||||
self._checkpoints: list[_Checkpoint] = []
|
||||
self._rows = 0
|
||||
self._byte_size = 0
|
||||
self._built = False
|
||||
|
||||
# ------------------------------------------------------------------ building
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
return self.directory / "forces.csv"
|
||||
|
||||
@property
|
||||
def rows(self) -> int:
|
||||
return self._rows
|
||||
|
||||
@property
|
||||
def byte_size(self) -> int:
|
||||
"""Size of the data file the index describes."""
|
||||
return self._byte_size
|
||||
|
||||
def build(self) -> "RecordingIndex":
|
||||
"""One streaming pass: times, flags, frame ids and byte offsets."""
|
||||
path = self.path
|
||||
times: list[float] = []
|
||||
valid: list[bool] = []
|
||||
frame_ids: list[int] = []
|
||||
checkpoints: list[_Checkpoint] = []
|
||||
if path.is_file():
|
||||
with path.open("rb") as handle:
|
||||
header = handle.readline().decode("utf-8", "replace").strip()
|
||||
columns = {name: index for index, name in enumerate(header.split(","))}
|
||||
time_column = columns.get("t_rel_s", 2)
|
||||
valid_column = columns.get("valid", 9)
|
||||
frame_column = columns.get("frame_id", 1)
|
||||
offset = handle.tell()
|
||||
for row, line in enumerate(handle):
|
||||
next_offset = handle.tell()
|
||||
text = line.decode("utf-8", "replace").rstrip("\r\n")
|
||||
if row % self.checkpoint_rows == 0:
|
||||
checkpoints.append(
|
||||
_Checkpoint(row=row, time=_time_of(text), offset=offset)
|
||||
)
|
||||
fields = next(csv.reader([text])) if text else []
|
||||
if len(fields) > max(time_column, valid_column, frame_column):
|
||||
try:
|
||||
times.append(float(fields[time_column]))
|
||||
valid.append(fields[valid_column].strip() == "1")
|
||||
frame_ids.append(int(fields[frame_column]))
|
||||
except (TypeError, ValueError):
|
||||
times.append(float("nan"))
|
||||
valid.append(False)
|
||||
frame_ids.append(-1)
|
||||
offset = next_offset
|
||||
self.times = np.asarray(times, dtype=np.float64)
|
||||
self.valid = np.asarray(valid, dtype=bool)
|
||||
self.frame_ids = np.asarray(frame_ids, dtype=np.int64)
|
||||
self._checkpoints = checkpoints
|
||||
self._rows = int(self.times.size)
|
||||
try:
|
||||
self._byte_size = path.stat().st_size
|
||||
except OSError:
|
||||
self._byte_size = 0
|
||||
self._built = True
|
||||
return self
|
||||
|
||||
def ensure_built(self) -> "RecordingIndex":
|
||||
if not self._built:
|
||||
self.build()
|
||||
return self
|
||||
|
||||
# ------------------------------------------------------------------- queries
|
||||
def extent(self) -> tuple[float, float]:
|
||||
if self._rows == 0:
|
||||
return (0.0, 0.0)
|
||||
return (float(self.times[0]), float(self.times[-1]))
|
||||
|
||||
def duration_s(self) -> float:
|
||||
first, last = self.extent()
|
||||
return max(0.0, last - first)
|
||||
|
||||
def row_range(self, t0: float, t1: float) -> tuple[int, int]:
|
||||
"""Half-open row range covering ``[t0, t1]``."""
|
||||
self.ensure_built()
|
||||
if self._rows == 0:
|
||||
return (0, 0)
|
||||
if float(t1) < float(t0):
|
||||
t0, t1 = t1, t0
|
||||
start = max(0, bisect_left(self.times.tolist(), float(t0)) - 1)
|
||||
stop = min(self._rows, bisect_right(self.times.tolist(), float(t1)))
|
||||
return (start, max(start, stop))
|
||||
|
||||
def _checkpoint_before(self, row: int) -> _Checkpoint:
|
||||
chosen = _Checkpoint(row=0, time=0.0, offset=0)
|
||||
for checkpoint in self._checkpoints:
|
||||
if checkpoint.row <= row:
|
||||
chosen = checkpoint
|
||||
else:
|
||||
break
|
||||
return chosen
|
||||
|
||||
def read_rows(
|
||||
self,
|
||||
start: int,
|
||||
stop: int,
|
||||
*,
|
||||
max_points: int | None = None,
|
||||
unit: str = "raw",
|
||||
) -> ForceWindow:
|
||||
"""Read ``[start, stop)`` by seeking to the nearest checkpoint.
|
||||
|
||||
With ``max_points`` the rows are decimated **while streaming**: each
|
||||
bucket contributes the extreme row of every axis in ``unit``, so a long
|
||||
interval never has to exist in memory in full. The extremes are taken in
|
||||
both the raw and the requested unit's axis, which is why a shear spike
|
||||
survives a flat normal channel.
|
||||
"""
|
||||
self.ensure_built()
|
||||
start = max(0, min(int(start), self._rows))
|
||||
stop = max(start, min(int(stop), self._rows))
|
||||
if stop <= start:
|
||||
return _empty_window()
|
||||
path = self.path
|
||||
if not path.is_file():
|
||||
return _empty_window()
|
||||
|
||||
newton_unit = str(unit).upper().startswith("N")
|
||||
budget = int(max_points) if max_points else 0
|
||||
total = stop - start
|
||||
bucket = (
|
||||
max(1, total // max(1, budget // 6))
|
||||
if budget and total > budget * 2
|
||||
else 1
|
||||
)
|
||||
|
||||
checkpoint = self._checkpoint_before(start)
|
||||
pending: list[tuple] = []
|
||||
picked: list[tuple] = []
|
||||
|
||||
def flush() -> None:
|
||||
if not pending:
|
||||
return
|
||||
if bucket == 1:
|
||||
picked.extend(pending)
|
||||
else:
|
||||
values = np.asarray(
|
||||
[row[3] if newton_unit else row[2] for row in pending],
|
||||
dtype=np.float64,
|
||||
)
|
||||
for axis in range(3):
|
||||
column = values[:, axis]
|
||||
finite = np.isfinite(column)
|
||||
if not finite.any():
|
||||
continue
|
||||
ordered = np.where(finite, column, -np.inf)
|
||||
for index in {int(np.argmin(ordered)), int(np.argmax(ordered))}:
|
||||
picked.append(pending[index])
|
||||
pending.clear()
|
||||
|
||||
with path.open("rb") as handle:
|
||||
# Every checkpoint offset points at a *data* row: the build records
|
||||
# ``tell()`` after the header, so there is nothing to skip here.
|
||||
handle.seek(checkpoint.offset)
|
||||
for row, line in enumerate(handle, start=checkpoint.row):
|
||||
if row >= stop:
|
||||
break
|
||||
if row < start:
|
||||
continue
|
||||
fields = next(csv.reader([line.decode("utf-8", "replace")]))
|
||||
if len(fields) < 11:
|
||||
continue
|
||||
try:
|
||||
t_rel = float(fields[2])
|
||||
frame_id = int(fields[1])
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
values = [_parse(fields[index]) for index in (3, 4, 5, 6, 7, 8)]
|
||||
pending.append(
|
||||
(
|
||||
frame_id,
|
||||
t_rel,
|
||||
tuple(values[:3]),
|
||||
tuple(values[3:]),
|
||||
fields[9].strip() == "1",
|
||||
)
|
||||
)
|
||||
if len(pending) >= bucket:
|
||||
flush()
|
||||
flush()
|
||||
|
||||
if not picked:
|
||||
return _empty_window()
|
||||
# A row can be the extreme of several axes; keep one copy, in time order.
|
||||
unique = {row[0]: row for row in picked}
|
||||
rows = sorted(unique.values(), key=lambda row: row[1])
|
||||
return ForceWindow(
|
||||
times=np.asarray([row[1] for row in rows], dtype=np.float64),
|
||||
raw=np.asarray([row[2] for row in rows], dtype=np.float64).reshape(-1, 3),
|
||||
newtons=np.asarray([row[3] for row in rows], dtype=np.float64).reshape(-1, 3),
|
||||
valid=np.asarray([row[4] for row in rows], dtype=bool),
|
||||
frame_ids=np.asarray([row[0] for row in rows], dtype=np.int64),
|
||||
decimated=bucket > 1,
|
||||
)
|
||||
|
||||
def iter_rows(self, start: int = 0, stop: Optional[int] = None) -> Iterator[tuple]:
|
||||
"""Yield ``(row, frame_id, t_rel_s, raw3, newtons3, valid)`` in file order."""
|
||||
self.ensure_built()
|
||||
path = self.path
|
||||
if not path.is_file():
|
||||
return
|
||||
stop = self._rows if stop is None else min(int(stop), self._rows)
|
||||
with path.open("rb") as handle:
|
||||
handle.readline() # header
|
||||
for row, line in enumerate(handle):
|
||||
if row >= stop:
|
||||
return
|
||||
if row < start:
|
||||
continue
|
||||
fields = next(csv.reader([line.decode("utf-8", "replace")]))
|
||||
if len(fields) < 11:
|
||||
continue
|
||||
try:
|
||||
frame_id = int(fields[1])
|
||||
t_rel = float(fields[2])
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
values = [_parse(fields[index]) for index in (3, 4, 5, 6, 7, 8)]
|
||||
yield (
|
||||
row,
|
||||
frame_id,
|
||||
t_rel,
|
||||
tuple(values[:3]),
|
||||
tuple(values[3:]),
|
||||
fields[9].strip() == "1",
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------ overview
|
||||
def overview(self, *, unit: str = "raw", buckets: int = DEFAULT_BUCKETS) -> Overview:
|
||||
"""A per-bucket envelope of the whole recording, in the active unit.
|
||||
|
||||
Built by streaming the file once and folding each row into its bucket, so
|
||||
a peak in a bucket the strip would otherwise skip is still visible.
|
||||
"""
|
||||
self.ensure_built()
|
||||
buckets = max(1, int(buckets))
|
||||
newton = str(unit).upper().startswith("N")
|
||||
if self._rows == 0:
|
||||
return Overview(
|
||||
times=np.empty(0),
|
||||
lows=np.empty((0, 3)),
|
||||
highs=np.empty((0, 3)),
|
||||
counts=np.empty(0, dtype=np.int64),
|
||||
unit="N" if newton else "raw",
|
||||
)
|
||||
first, last = self.extent()
|
||||
span = max(1e-9, last - first)
|
||||
lows = np.full((buckets, 3), np.nan)
|
||||
highs = np.full((buckets, 3), np.nan)
|
||||
counts = np.zeros(buckets, dtype=np.int64)
|
||||
starts = np.full(buckets, np.nan)
|
||||
for _row, _frame_id, t_rel, raw, newtons, is_valid in self.iter_rows():
|
||||
if not is_valid:
|
||||
continue
|
||||
bucket = int((t_rel - first) / span * (buckets - 1) + 0.5)
|
||||
bucket = max(0, min(buckets - 1, bucket))
|
||||
values = newtons if newton else raw
|
||||
for axis in range(3):
|
||||
value = values[axis]
|
||||
if value != value: # NaN
|
||||
continue
|
||||
current_low = lows[bucket, axis]
|
||||
current_high = highs[bucket, axis]
|
||||
lows[bucket, axis] = value if current_low != current_low else min(current_low, value)
|
||||
highs[bucket, axis] = value if current_high != current_high else max(current_high, value)
|
||||
if starts[bucket] != starts[bucket]:
|
||||
starts[bucket] = t_rel
|
||||
counts[bucket] += 1
|
||||
return Overview(
|
||||
times=starts,
|
||||
lows=lows,
|
||||
highs=highs,
|
||||
counts=counts,
|
||||
unit="N" if newton else "raw",
|
||||
)
|
||||
|
||||
|
||||
def _time_of(text: str) -> float:
|
||||
fields = next(csv.reader([text])) if text else []
|
||||
if len(fields) > 2:
|
||||
try:
|
||||
return float(fields[2])
|
||||
except (TypeError, ValueError):
|
||||
return float("nan")
|
||||
return float("nan")
|
||||
|
||||
|
||||
def _empty_window() -> ForceWindow:
|
||||
return ForceWindow(
|
||||
times=np.empty(0, dtype=np.float64),
|
||||
raw=np.empty((0, 3)),
|
||||
newtons=np.empty((0, 3)),
|
||||
valid=np.empty(0, dtype=bool),
|
||||
frame_ids=np.empty(0, dtype=np.int64),
|
||||
)
|
||||
@@ -21,6 +21,15 @@ from typing import Iterator, Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
from orisys_app_shared.record.index import (
|
||||
CHECKPOINT_ROWS,
|
||||
DEFAULT_BUCKETS,
|
||||
ForceWindow,
|
||||
Overview,
|
||||
RecordingIndex,
|
||||
)
|
||||
from orisys_app_shared.record.sample import plottable_axis_mask
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Column names as written by the recorder (index == position in forces.csv).
|
||||
@@ -65,9 +74,21 @@ class RecordingInfo:
|
||||
frame_ids: np.ndarray
|
||||
dropped_rows: int = 0
|
||||
frames_dropped: int = 0
|
||||
#: The index rows were in time order as written (False = a legacy record).
|
||||
frames_out_of_order: bool = True
|
||||
#: The force rows were in time order as written.
|
||||
forces_out_of_order: bool = True
|
||||
#: True when the arrays above are a decimated view of a long recording.
|
||||
decimated: bool = False
|
||||
#: Total rows in ``forces.csv`` (the decimated view has fewer).
|
||||
total_rows: int = 0
|
||||
#: The recording's real duration, from the frame index, never from the view.
|
||||
total_duration_s: float = 0.0
|
||||
|
||||
@property
|
||||
def duration_s(self) -> float:
|
||||
if self.total_duration_s:
|
||||
return float(self.total_duration_s)
|
||||
if self.times.size == 0:
|
||||
return 0.0
|
||||
return float(self.times[-1])
|
||||
@@ -78,26 +99,51 @@ class RecordingInfo:
|
||||
return self.raw
|
||||
|
||||
def plottable(self, unit: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""``(times, values, mask)`` — the same validity rule the live charts use.
|
||||
"""``(times, values, mask)`` — the same rule the live charts use.
|
||||
|
||||
The replay used to filter on finiteness alone, so a stale value left by a
|
||||
failed compute step could reappear here after another view had already
|
||||
excluded it.
|
||||
excluded it. ``mask`` is per axis: a single-axis calibration is normal and
|
||||
must not hide the axis that *is* calibrated.
|
||||
"""
|
||||
values = self.value_for_unit(unit)
|
||||
mask = self.valid & np.isfinite(values).all(axis=1)
|
||||
return self.times[mask], values[mask], mask
|
||||
return self.times, values, plottable_axis_mask(values, self.valid)
|
||||
|
||||
def plottable_axis(self, unit: str, index: int) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""``(times, values)`` for one channel, with the other axes ignored."""
|
||||
values = self.value_for_unit(unit)
|
||||
mask = plottable_axis_mask(values, self.valid)[:, index]
|
||||
return self.times[mask], values[mask, index]
|
||||
|
||||
|
||||
class RecordingReader:
|
||||
"""Open one experiment directory and serve replay queries from it."""
|
||||
"""Open one experiment directory and serve replay queries from it.
|
||||
|
||||
def __init__(self, directory: str | Path) -> None:
|
||||
``max_rows`` bounds what :meth:`info` keeps in memory: a long recording is
|
||||
loaded as a decimated view (peaks preserved) and marked ``decimated``, while
|
||||
:meth:`window` and :meth:`overview` read the interval being drawn from disk
|
||||
through :class:`~orisys_app_shared.record.index.RecordingIndex`. That is what
|
||||
lets the replay show an hour-long recording without holding all of it.
|
||||
"""
|
||||
|
||||
#: Above this many rows ``info()`` decimates instead of loading everything.
|
||||
DEFAULT_MAX_ROWS = 200_000
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
directory: str | Path,
|
||||
*,
|
||||
max_rows: int | None = None,
|
||||
checkpoint_rows: int = CHECKPOINT_ROWS,
|
||||
) -> None:
|
||||
self.directory = Path(directory)
|
||||
if not self.directory.is_dir():
|
||||
raise FileNotFoundError(f"Not a recording directory: {self.directory}")
|
||||
self._info: Optional[RecordingInfo] = None
|
||||
self._events: Optional[list[dict]] = None
|
||||
self.max_rows = int(max_rows or self.DEFAULT_MAX_ROWS)
|
||||
self._index = RecordingIndex(self.directory, checkpoint_rows=checkpoint_rows)
|
||||
self._overviews: dict[tuple[str, int], Overview] = {}
|
||||
|
||||
# ------------------------------------------------------------------ loading
|
||||
def info(self) -> RecordingInfo:
|
||||
@@ -105,6 +151,10 @@ class RecordingReader:
|
||||
self._info = self._load()
|
||||
return self._info
|
||||
|
||||
def index(self) -> RecordingIndex:
|
||||
"""The time index, built once and shared with the windows."""
|
||||
return self._index.ensure_built()
|
||||
|
||||
def session(self) -> dict:
|
||||
path = self.directory / "session.json"
|
||||
if not path.is_file():
|
||||
@@ -116,8 +166,30 @@ class RecordingReader:
|
||||
return {}
|
||||
|
||||
def _load(self) -> RecordingInfo:
|
||||
frames = self._load_frames()
|
||||
times, raw, newtons, valid, frame_ids, dropped = self._load_forces()
|
||||
frames, frames_ordered = self._load_frames()
|
||||
index = self.index()
|
||||
total_rows = index.rows
|
||||
decimated = total_rows > self.max_rows > 0
|
||||
if decimated:
|
||||
window = self.window_all(
|
||||
max_points=self.max_rows
|
||||
)
|
||||
times = window.times
|
||||
raw = window.raw
|
||||
newtons = window.newtons
|
||||
valid = window.valid
|
||||
frame_ids = window.frame_ids
|
||||
forces_ordered = True
|
||||
dropped = 0
|
||||
else:
|
||||
(times, raw, newtons, valid, frame_ids, dropped, forces_ordered) = (
|
||||
self._load_forces()
|
||||
)
|
||||
total_duration = 0.0
|
||||
if frames:
|
||||
total_duration = float(frames[-1].t_rel_s)
|
||||
elif times.size:
|
||||
total_duration = float(times[-1])
|
||||
return RecordingInfo(
|
||||
directory=self.directory,
|
||||
session=self.session(),
|
||||
@@ -129,12 +201,26 @@ class RecordingReader:
|
||||
frame_ids=frame_ids,
|
||||
dropped_rows=dropped,
|
||||
frames_dropped=sum(1 for frame in frames if frame.dropped),
|
||||
frames_out_of_order=frames_ordered,
|
||||
forces_out_of_order=forces_ordered,
|
||||
decimated=decimated,
|
||||
total_rows=total_rows,
|
||||
total_duration_s=total_duration,
|
||||
)
|
||||
|
||||
def _load_frames(self) -> list[RecordingFrame]:
|
||||
def _load_frames(self) -> tuple[list[RecordingFrame], bool]:
|
||||
"""Load the frame index in time order.
|
||||
|
||||
Recordings made before the ordered-write fix (see
|
||||
``ExperimentRecorder``) can contain a frame index whose rows are not in
|
||||
time order, because the row backlog used to be drained before the queue.
|
||||
``frame_at`` does a binary search over these times, so an unsorted index
|
||||
would silently answer with the wrong frame. The rows are sorted here and
|
||||
the fact is reported as a replay warning.
|
||||
"""
|
||||
path = self.directory / "frames.csv"
|
||||
if not path.is_file():
|
||||
return []
|
||||
return [], False
|
||||
rows: list[RecordingFrame] = []
|
||||
with path.open(encoding="utf-8", newline="") as handle:
|
||||
for row in csv.DictReader(handle):
|
||||
@@ -152,13 +238,26 @@ class RecordingReader:
|
||||
)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
continue
|
||||
return rows
|
||||
ordered = _is_time_ordered([frame.t_rel_s for frame in rows])
|
||||
if not ordered:
|
||||
rows.sort(key=lambda frame: frame.t_rel_s)
|
||||
return rows, ordered
|
||||
|
||||
def _load_forces(self):
|
||||
path = self.directory / "forces.csv"
|
||||
if not path.is_file():
|
||||
# A directory with no data (an empty or half-written recording) is
|
||||
# readable: it simply has no samples, and nothing is out of order.
|
||||
empty = np.empty(0, dtype=np.float64)
|
||||
return empty, np.empty((0, 3)), np.empty((0, 3)), np.empty(0, bool), np.empty(0, int), 0
|
||||
return (
|
||||
empty,
|
||||
np.empty((0, 3)),
|
||||
np.empty((0, 3)),
|
||||
np.empty(0, bool),
|
||||
np.empty(0, int),
|
||||
0,
|
||||
True,
|
||||
)
|
||||
|
||||
def parse(text: str) -> float:
|
||||
text = (text or "").strip()
|
||||
@@ -191,23 +290,42 @@ class RecordingReader:
|
||||
(parse(row.get("FN_N")), parse(row.get("FX_N")), parse(row.get("FY_N")))
|
||||
)
|
||||
valid.append((row.get("valid") or "0").strip() == "1")
|
||||
return (
|
||||
np.asarray(times, dtype=np.float64),
|
||||
np.asarray(raw, dtype=np.float64).reshape(-1, 3),
|
||||
np.asarray(newtons, dtype=np.float64).reshape(-1, 3),
|
||||
np.asarray(valid, dtype=bool),
|
||||
np.asarray(frame_ids, dtype=np.int64),
|
||||
dropped,
|
||||
)
|
||||
times_arr = np.asarray(times, dtype=np.float64)
|
||||
raw_arr = np.asarray(raw, dtype=np.float64).reshape(-1, 3)
|
||||
newtons_arr = np.asarray(newtons, dtype=np.float64).reshape(-1, 3)
|
||||
valid_arr = np.asarray(valid, dtype=bool)
|
||||
ids_arr = np.asarray(frame_ids, dtype=np.int64)
|
||||
|
||||
# Older recordings can have rows out of time order (the writer used to
|
||||
# drain a backlog before the queue). Every window query below assumes
|
||||
# sorted time, so sort once here and report it instead of answering
|
||||
# ``searchsorted`` on unsorted input.
|
||||
ordered = _is_time_ordered(times)
|
||||
if not ordered and times_arr.size:
|
||||
order = np.argsort(times_arr, kind="stable")
|
||||
times_arr = times_arr[order]
|
||||
raw_arr = raw_arr[order]
|
||||
newtons_arr = newtons_arr[order]
|
||||
valid_arr = valid_arr[order]
|
||||
ids_arr = ids_arr[order]
|
||||
return times_arr, raw_arr, newtons_arr, valid_arr, ids_arr, dropped, ordered
|
||||
|
||||
# ------------------------------------------------------------------ queries
|
||||
def frame_at(self, t_rel_s: float) -> Optional[RecordingFrame]:
|
||||
"""The recorded frame at or just before ``t_rel_s``."""
|
||||
"""The recorded frame at or just before ``t_rel_s``.
|
||||
|
||||
The index is sorted at load time, so the binary search is valid; the
|
||||
fallback walk keeps a hand-edited or truncated file from answering with
|
||||
the wrong frame.
|
||||
"""
|
||||
frames = self.info().frames
|
||||
if not frames:
|
||||
return None
|
||||
times = [frame.t_rel_s for frame in frames]
|
||||
index = int(np.searchsorted(np.asarray(times), float(t_rel_s), side="right")) - 1
|
||||
times = np.asarray([frame.t_rel_s for frame in frames], dtype=np.float64)
|
||||
if not _is_time_ordered(times.tolist()):
|
||||
candidates = [frame for frame in frames if frame.t_rel_s <= float(t_rel_s)]
|
||||
return max(candidates, key=lambda frame: frame.t_rel_s) if candidates else frames[0]
|
||||
index = int(np.searchsorted(times, float(t_rel_s), side="right")) - 1
|
||||
index = max(0, min(index, len(frames) - 1))
|
||||
return frames[index]
|
||||
|
||||
@@ -318,6 +436,47 @@ class RecordingReader:
|
||||
mask = (info.times >= float(t0)) & (info.times <= float(t1))
|
||||
return info.times[mask], info.raw[mask]
|
||||
|
||||
# ------------------------------------------------------ on-demand history
|
||||
def window(
|
||||
self,
|
||||
t0: float,
|
||||
t1: float,
|
||||
*,
|
||||
unit: str = "raw",
|
||||
max_points: int | None = None,
|
||||
) -> ForceWindow:
|
||||
"""The recorded rows inside ``[t0, t1]``, read from disk.
|
||||
|
||||
This is the interval read: it seeks to the nearest checkpoint in the
|
||||
index and scans forward, so asking for ten seconds of an hour-long
|
||||
recording costs ten seconds of parsing, not an hour. ``max_points``
|
||||
decimates *while* reading, so the interval never has to exist in memory
|
||||
in full.
|
||||
"""
|
||||
start, stop = self.index().row_range(t0, t1)
|
||||
return self.index().read_rows(
|
||||
start, stop, max_points=max_points, unit=unit
|
||||
)
|
||||
|
||||
def window_all(self, *, unit: str = "raw", max_points: int | None = None) -> ForceWindow:
|
||||
"""The whole recording as one window, decimated while streaming."""
|
||||
return self.window(
|
||||
*self.index().extent(), unit=unit, max_points=max_points
|
||||
)
|
||||
|
||||
def overview(self, *, unit: str = "raw", buckets: int = DEFAULT_BUCKETS) -> Overview:
|
||||
"""A whole-session envelope, cached per (unit, bucket count)."""
|
||||
key = (str(unit), int(buckets))
|
||||
if key not in self._overviews:
|
||||
self._overviews[key] = self.index().overview(
|
||||
unit=str(unit), buckets=int(buckets)
|
||||
)
|
||||
return self._overviews[key]
|
||||
|
||||
def statistics(self, t0: float, t1: float, *, unit: str = "raw"):
|
||||
"""Statistics of an interval, computed from the interval's own rows."""
|
||||
return self.window(t0, t1, unit=unit).statistics(unit)
|
||||
|
||||
# ------------------------------------------------------------------ policy
|
||||
def depth_policy(self) -> dict:
|
||||
return dict(self.info().session.get("depth") or {})
|
||||
@@ -372,6 +531,8 @@ class RecordingReader:
|
||||
warnings.append("rows_skipped")
|
||||
if self.info().frames_dropped:
|
||||
warnings.append("frames_dropped")
|
||||
if not self.info().frames_out_of_order or not self.info().forces_out_of_order:
|
||||
warnings.append("out_of_order")
|
||||
return warnings
|
||||
|
||||
def iter_forces(self, batch: int = 4096) -> Iterator[tuple[np.ndarray, np.ndarray]]:
|
||||
@@ -400,8 +561,56 @@ class RecordingReader:
|
||||
yield np.asarray(times), np.asarray(values).reshape(-1, 6)
|
||||
|
||||
|
||||
def _is_time_ordered(values) -> bool:
|
||||
"""True when ``values`` never decreases (an empty or single value is ordered)."""
|
||||
previous = None
|
||||
for value in values:
|
||||
if previous is not None and float(value) < float(previous):
|
||||
return False
|
||||
previous = value
|
||||
return True
|
||||
|
||||
|
||||
def decimate_window(window: ForceWindow, max_points: int, *, unit: str = "raw") -> ForceWindow:
|
||||
"""Reduce a window to about ``max_points``, keeping every axis' extremes.
|
||||
|
||||
The same rule the live store uses: each bucket contributes the extreme row of
|
||||
**every plotted axis** in the active unit, so a shear spike survives a flat
|
||||
normal channel. The result is marked ``decimated``.
|
||||
"""
|
||||
total = len(window)
|
||||
max_points = max(4, int(max_points))
|
||||
if total <= max_points:
|
||||
return window
|
||||
values = window.value_for_unit(unit)
|
||||
def index_of(row: int) -> int:
|
||||
return int(window.frame_ids[row]) if total else row
|
||||
bucket = max(1, total // max(1, max_points // 6))
|
||||
picked: set[int] = set()
|
||||
for start in range(0, total, bucket):
|
||||
stop = min(total, start + bucket)
|
||||
for axis in range(3):
|
||||
column = values[start:stop, axis]
|
||||
finite = np.isfinite(column)
|
||||
if not finite.any():
|
||||
continue
|
||||
ordered = np.where(finite, column, -np.inf)
|
||||
picked.add(start + int(np.argmin(ordered)))
|
||||
picked.add(start + int(np.argmax(ordered)))
|
||||
if not picked:
|
||||
return window
|
||||
rows = sorted(picked)
|
||||
return ForceWindow(
|
||||
times=window.times[rows],
|
||||
raw=window.raw[rows],
|
||||
newtons=window.newtons[rows],
|
||||
valid=window.valid[rows],
|
||||
frame_ids=window.frame_ids[rows],
|
||||
decimated=True,
|
||||
)
|
||||
|
||||
|
||||
def list_recordings(root: str | Path) -> list[Path]:
|
||||
"""Every recording directory under ``root``, newest first."""
|
||||
base = Path(root)
|
||||
if not base.is_dir():
|
||||
return []
|
||||
|
||||
@@ -140,11 +140,16 @@ def to_record_sample(packet, *, calibration_version: str = "") -> RecordSample:
|
||||
valid = bool(getattr(packet, "valid", True))
|
||||
reason = str(getattr(packet, "invalid_reason", "") or "")
|
||||
stale = bool(getattr(packet, "stale_forces", False))
|
||||
# The packet's own version wins: it was stamped by the acquisition thread at
|
||||
# the moment of acquisition, so it is the only value that stays true when a
|
||||
# calibration is applied in the middle of a session.
|
||||
version = str(
|
||||
getattr(packet, "calibration_version", "") or calibration_version or ""
|
||||
# A packet that *carries* a version owns it, including the empty one: the
|
||||
# acquisition thread stamps every frame, and an empty stamp is the device
|
||||
# saying "no calibration loaded". Falling back to the session default there
|
||||
# would resurrect coefficients that are no longer in effect. A packet from an
|
||||
# older producer that has no version field at all still gets the default.
|
||||
packet_version = getattr(packet, "calibration_version", None)
|
||||
version = (
|
||||
str(calibration_version or "")
|
||||
if packet_version is None
|
||||
else str(packet_version)
|
||||
)
|
||||
axes = calibrated_axes_of(packet)
|
||||
|
||||
@@ -214,3 +219,20 @@ def plottable_mask(values: Iterable[float], valid: Iterable[bool]) -> list[bool]
|
||||
bool(is_valid) and math.isfinite(float(value))
|
||||
for value, is_valid in zip(values, valid)
|
||||
]
|
||||
|
||||
|
||||
def plottable_axis_mask(values, valid) -> "np.ndarray":
|
||||
"""``(n, axes)`` mask: one column per axis, decided for that axis alone.
|
||||
|
||||
The rule has to be per axis. A single-axis calibration is legitimate — the
|
||||
device may have a normal-force curve and no shear curves — and an all-axes
|
||||
rule hid a perfectly good FN because FX and FY had no newton value.
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
block = np.asarray(values, dtype=np.float64).reshape(-1, 3)
|
||||
validity = np.asarray(valid, dtype=bool).reshape(-1)
|
||||
finite = np.isfinite(block)
|
||||
if validity.size != block.shape[0]:
|
||||
return finite
|
||||
return finite & validity[:, None]
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""The video side of a recording: one rolling MP4 segment at a time.
|
||||
|
||||
Kept apart from the CSV writer on purpose. The two have different failure modes —
|
||||
a full disk breaks both, but a codec that refuses to open only breaks this one —
|
||||
and the recorder's byte budget is about *these* frames, not about the rows.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import cv2
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class VideoSegments:
|
||||
"""Writes raw frames into ``raw_NNNN.mp4`` files, rolling on a time budget."""
|
||||
|
||||
def __init__(self, directory: Path, *, codec: str, fps: float, segment_seconds: float):
|
||||
self.directory = Path(directory)
|
||||
self.codec = str(codec)
|
||||
self.fps = max(1.0, float(fps))
|
||||
self.segment_seconds = float(segment_seconds)
|
||||
self.segments = 0
|
||||
self.video_frames = 0
|
||||
self.frame_shape: Optional[tuple[int, int]] = None
|
||||
self._writer: Optional[cv2.VideoWriter] = None
|
||||
self._index = 0
|
||||
self._frames_in_segment = 0
|
||||
self._started = 0.0
|
||||
|
||||
# ------------------------------------------------------------------ writing
|
||||
def write(self, frame) -> tuple[int, int]:
|
||||
"""Append one frame. Returns ``(segment, frame_in_segment)``."""
|
||||
if self.frame_shape is None:
|
||||
self.frame_shape = (int(frame.shape[1]), int(frame.shape[0]))
|
||||
if self._should_roll():
|
||||
self._close_segment()
|
||||
if self._writer is None:
|
||||
self._open_segment()
|
||||
self._writer.write(frame)
|
||||
self._frames_in_segment += 1
|
||||
self.video_frames += 1
|
||||
return (self._index, self._frames_in_segment)
|
||||
|
||||
def write_sequence(self, frames) -> int:
|
||||
"""Write a run of frames as one standalone segment. Returns the count."""
|
||||
frames = [frame for frame in frames if frame is not None]
|
||||
if not frames:
|
||||
return 0
|
||||
self.frame_shape = (int(frames[0].shape[1]), int(frames[0].shape[0]))
|
||||
name = "preroll_0001.mp4"
|
||||
writer = self._open_writer(self.directory / name)
|
||||
written = 0
|
||||
try:
|
||||
for frame in frames:
|
||||
writer.write(frame)
|
||||
written += 1
|
||||
finally:
|
||||
writer.release()
|
||||
return written
|
||||
|
||||
# ----------------------------------------------------------------- segments
|
||||
def _should_roll(self) -> bool:
|
||||
if self._writer is None:
|
||||
return False
|
||||
if self.segment_seconds <= 0:
|
||||
return False
|
||||
return (time.perf_counter() - self._started) >= self.segment_seconds
|
||||
|
||||
def _open_writer(self, path: Path) -> cv2.VideoWriter:
|
||||
fourcc = cv2.VideoWriter_fourcc(*self.codec)
|
||||
writer = cv2.VideoWriter(str(path), fourcc, self.fps, self.frame_shape)
|
||||
if not writer.isOpened():
|
||||
raise OSError(f"could not open video writer for {path.name}")
|
||||
return writer
|
||||
|
||||
def _open_segment(self) -> None:
|
||||
self._index += 1
|
||||
path = self.directory / f"raw_{self._index:04d}.mp4"
|
||||
self._writer = self._open_writer(path)
|
||||
self._frames_in_segment = 0
|
||||
self._started = time.perf_counter()
|
||||
self.segments = self._index
|
||||
|
||||
def _close_segment(self) -> None:
|
||||
if self._writer is None:
|
||||
return
|
||||
self._writer.release()
|
||||
self._writer = None
|
||||
|
||||
def close(self) -> None:
|
||||
self._close_segment()
|
||||
@@ -0,0 +1,280 @@
|
||||
"""Calibration identity and operator events as the acquisition thread sees them.
|
||||
|
||||
The review's P1-7: the application event was written before the device had loaded
|
||||
anything, the recorded ``calibration_id`` never changed mid-session, and an
|
||||
uncalibrated device could keep claiming the coefficients of an earlier session.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkAcquisitionWorker
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig
|
||||
from orisys_app_shared.record.calibration_snapshot import (
|
||||
list_calibration_snapshots,
|
||||
snapshot_name,
|
||||
store_calibration_snapshot,
|
||||
)
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
|
||||
def _root() -> Path:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
return SCRATCH
|
||||
|
||||
|
||||
class _Sensor:
|
||||
"""A device with a swappable calibration, like the SDK's Sensor."""
|
||||
|
||||
def __init__(self, calibration=None, path=None):
|
||||
self.calibration = calibration
|
||||
self.calibration_path = path
|
||||
self.applied: list[str] = []
|
||||
self.resets = 0
|
||||
|
||||
def set_calibration(self, path):
|
||||
if str(path) == "reject-me":
|
||||
raise ValueError("bad document")
|
||||
self.calibration = object()
|
||||
self.calibration_path = Path(path)
|
||||
self.applied.append(str(path))
|
||||
|
||||
def reset(self):
|
||||
self.resets += 1
|
||||
|
||||
|
||||
def _calibration_file(root: Path, name: str, body: str = '{"unit": "N"}') -> Path:
|
||||
path = root / name
|
||||
path.write_text(body, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _worker(**kwargs) -> SdkAcquisitionWorker:
|
||||
for key in ("on_calibration_applied", "on_reset_applied"):
|
||||
kwargs.setdefault(key, None)
|
||||
return SdkAcquisitionWorker(**kwargs)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ worker state
|
||||
|
||||
|
||||
def test_a_new_session_does_not_inherit_the_previous_version():
|
||||
"""An uncalibrated device must report no calibration, not the last one."""
|
||||
root = _root()
|
||||
worker = _worker()
|
||||
worker._calibration_version = "previous-session-hash"
|
||||
worker._stop.set()
|
||||
worker.start("clip.mp4", config_ref=str(_calibration_file(root, "sensor.json")))
|
||||
assert worker._calibration_version == "", (
|
||||
"the new session kept the previous session's calibration version"
|
||||
)
|
||||
worker.stop(wait=True)
|
||||
|
||||
|
||||
def test_the_applied_version_is_the_content_hash_and_carries_the_frame():
|
||||
root = _root()
|
||||
applied: list[tuple] = []
|
||||
worker = _worker(on_calibration_applied=lambda *args: applied.append(args))
|
||||
path = _calibration_file(root, "force_calibration.json")
|
||||
sensor = _Sensor()
|
||||
worker._session_id = 4
|
||||
worker.set_pending_calibration(path)
|
||||
worker._consume_pending_calibration(sensor, frame_id=17, mono_t=0.5)
|
||||
|
||||
assert len(applied) == 1
|
||||
got_path, version, frame_id, mono_t, session = applied[0]
|
||||
assert got_path == str(path)
|
||||
assert version and version == worker._calibration_version
|
||||
assert version != str(path), "the path was used as the version"
|
||||
assert (frame_id, mono_t, session) == (17, 0.5, 4)
|
||||
assert sensor.applied == [str(path)]
|
||||
|
||||
|
||||
def test_a_rejected_calibration_keeps_the_version_the_device_still_has():
|
||||
root = _root()
|
||||
applied: list[tuple] = []
|
||||
worker = _worker(on_calibration_applied=lambda *args: applied.append(args))
|
||||
sensor = _Sensor()
|
||||
worker._session_id = 1
|
||||
worker._calibration_version = "loaded-earlier"
|
||||
worker.set_pending_calibration("reject-me")
|
||||
with pytest.raises(ValueError):
|
||||
worker._consume_pending_calibration(sensor, frame_id=3, mono_t=0.1)
|
||||
assert not applied, "a rejected document was reported as applied"
|
||||
assert worker._calibration_version == "loaded-earlier", (
|
||||
"the device still has those coefficients, so the version is still true"
|
||||
)
|
||||
|
||||
|
||||
def test_a_zeroing_is_applied_by_the_worker_and_reported_with_its_frame():
|
||||
root = _root()
|
||||
events: list[tuple] = []
|
||||
worker = _worker(on_reset_applied=lambda *args: events.append(args))
|
||||
sensor = _Sensor()
|
||||
worker._session_id = 2
|
||||
worker.reset() # a request, not the zeroing itself
|
||||
assert sensor.resets == 0, "the zeroing ran on the caller's thread"
|
||||
worker._consume_pending_reset(sensor, frame_id=42, mono_t=1.4)
|
||||
assert sensor.resets == 1
|
||||
assert events == [(42, 1.4, 2)]
|
||||
# The request is consumed once.
|
||||
worker._consume_pending_reset(sensor, frame_id=43, mono_t=1.5)
|
||||
assert events == [(42, 1.4, 2)]
|
||||
|
||||
|
||||
def test_a_started_session_drops_a_stale_zeroing_request():
|
||||
root = _root()
|
||||
events: list[tuple] = []
|
||||
worker = _worker(on_reset_applied=lambda *args: events.append(args))
|
||||
worker._stop.set()
|
||||
worker.start("clip.mp4", config_ref=str(_calibration_file(root, "sensor.json")))
|
||||
sensor = _Sensor()
|
||||
worker._consume_pending_reset(sensor, frame_id=1, mono_t=0.0)
|
||||
assert sensor.resets == 0, "a reset requested before this session fired into it"
|
||||
assert not events
|
||||
worker.stop(wait=True)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- snapshots
|
||||
|
||||
|
||||
def test_a_snapshot_is_content_addressed_and_listed():
|
||||
root = _root()
|
||||
directory = root / "session"
|
||||
source = _calibration_file(root, "force_calibration_box.json", '{"a": 1}')
|
||||
name = store_calibration_snapshot(directory, source, "abcdef1234567890")
|
||||
assert name == "calibration/force_calibration_abcdef123456.json"
|
||||
assert (directory / name).is_file()
|
||||
assert (directory / name).read_text(encoding="utf-8") == '{"a": 1}'
|
||||
# Applying the same coefficients twice stores one file.
|
||||
again = store_calibration_snapshot(directory, source, "abcdef1234567890")
|
||||
assert again == name
|
||||
assert list_calibration_snapshots(directory) == [name]
|
||||
# A different calibration cannot collide with it.
|
||||
other = _calibration_file(root, "force_calibration_box.json", '{"a": 2}')
|
||||
second = store_calibration_snapshot(directory, other, "ffffff000000")
|
||||
assert second != name
|
||||
assert len(list_calibration_snapshots(directory)) == 2
|
||||
assert snapshot_name("", source).startswith("force_calibration_")
|
||||
|
||||
|
||||
def test_the_recording_metadata_names_the_coefficients_it_ran_with():
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
source = _calibration_file(root, "force_calibration_box.json", '{"a": 1}')
|
||||
name = store_calibration_snapshot(directory, source, "abcdef1234567890")
|
||||
recorder.note_calibration_snapshot(
|
||||
version="abcdef1234567890", name=name, frame_id=12, mono_t=0.4
|
||||
)
|
||||
recorder.log_event(
|
||||
"calibration_applied", name, mono_t=0.4, frame_id=12
|
||||
)
|
||||
recorder.stop()
|
||||
|
||||
meta = json.loads((Path(directory) / "session.json").read_text(encoding="utf-8"))
|
||||
snapshots = meta["calibration_snapshots"]
|
||||
assert snapshots and snapshots[0]["version"] == "abcdef1234567890"
|
||||
assert snapshots[0]["file"] == name
|
||||
assert snapshots[0]["frame_id"] == 12
|
||||
assert snapshots[0]["mono_t"] == pytest.approx(0.4)
|
||||
event = next(e for e in meta["events"] if e["kind"] == "calibration_applied")
|
||||
assert event["frame_id"] == 12
|
||||
assert event["mono_t"] == pytest.approx(0.4), "the event lost its acquisition time"
|
||||
assert event["at"], "the event lost its wall clock"
|
||||
|
||||
# The CSV carries the same acquisition time in the session's relative base.
|
||||
# An event that predates the first frame has no base yet, so its relative
|
||||
# time is empty — the absolute (session-relative) time is in the metadata.
|
||||
with (Path(directory) / "events.csv").open(encoding="utf-8", newline="") as handle:
|
||||
import csv as _csv
|
||||
|
||||
rows = list(_csv.DictReader(handle))
|
||||
applied = next(row for row in rows if row["kind"] == "calibration_applied")
|
||||
assert applied["frame_id"] == "12"
|
||||
assert applied["t_rel_s"] == "", "an event before the first frame invented a base"
|
||||
|
||||
|
||||
def test_an_event_during_the_recording_gets_the_session_relative_time():
|
||||
import csv as _csv
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
frame = np.zeros((4, 4, 3), np.uint8)
|
||||
recorder.on_packet(
|
||||
SdkPacket(
|
||||
raw_frame=frame,
|
||||
processed_frame=frame,
|
||||
timestamp=1000.0,
|
||||
mono_t=0.5,
|
||||
fps=30.0,
|
||||
process_ms=1.0,
|
||||
frame_id=7,
|
||||
)
|
||||
)
|
||||
recorder.log_event("zeroing", "tracker reset", mono_t=0.9, frame_id=9)
|
||||
recorder.stop()
|
||||
|
||||
with (Path(directory) / "events.csv").open(encoding="utf-8", newline="") as handle:
|
||||
rows = list(_csv.DictReader(handle))
|
||||
zeroing = next(row for row in rows if row["kind"] == "zeroing")
|
||||
assert float(zeroing["t_rel_s"]) == pytest.approx(0.4, abs=1e-6)
|
||||
assert zeroing["frame_id"] == "9"
|
||||
|
||||
|
||||
def test_an_event_without_a_mono_time_still_gets_a_wall_clock():
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
recorder.log_event("operator_note", "no frame attached")
|
||||
recorder.stop()
|
||||
meta = json.loads((Path(directory) / "session.json").read_text(encoding="utf-8"))
|
||||
event = next(e for e in meta["events"] if e["kind"] == "operator_note")
|
||||
assert event["at"], "the event lost its wall clock"
|
||||
assert event["frame_id"] == -1
|
||||
|
||||
|
||||
def test_the_recorded_row_uses_the_packet_version_not_the_session_default():
|
||||
from orisys_app_shared.record.sample import to_record_sample
|
||||
|
||||
class Packet:
|
||||
frame_id = 3
|
||||
mono_t = 0.2
|
||||
wall_t = 0.2
|
||||
valid = True
|
||||
calibration_version = ""
|
||||
calibrated_axes = ("fn",)
|
||||
force_calibrated = True
|
||||
raw_available = True
|
||||
|
||||
# An empty packet version is the device saying "no calibration": the session
|
||||
# default must not be pasted in behind its back.
|
||||
assert to_record_sample(Packet(), calibration_version="session-default").calibration_version == ""
|
||||
|
||||
class LegacyPacket(Packet):
|
||||
calibration_version = None # an older producer without the field
|
||||
|
||||
# ...but a packet that has no version field at all still gets the default.
|
||||
assert (
|
||||
to_record_sample(LegacyPacket(), calibration_version="session-default").calibration_version
|
||||
== "session-default"
|
||||
)
|
||||
|
||||
class StampedPacket(Packet):
|
||||
calibration_version = "packet-hash"
|
||||
|
||||
assert (
|
||||
to_record_sample(StampedPacket(), calibration_version="session-default").calibration_version
|
||||
== "packet-hash"
|
||||
)
|
||||
@@ -8,6 +8,9 @@ model.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.force.calibration_io import (
|
||||
@@ -17,10 +20,19 @@ from orisys_app_shared.force.calibration_io import (
|
||||
build_force_calibration_v2,
|
||||
merge_samples,
|
||||
predict_channel,
|
||||
write_force_calibration,
|
||||
)
|
||||
from orisys_app_shared.force.calibration_recorder import CalibrationRecorder
|
||||
from orisys_app_shared.force.session import SessionSample
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
|
||||
def _root() -> Path:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
return SCRATCH
|
||||
|
||||
|
||||
def _point(recorder, load: float, raw: float, *, channel: str = "fn"):
|
||||
recorder.start(channel=channel, reference_load_n=load, duration_s=0.05)
|
||||
@@ -151,3 +163,84 @@ def test_a_channel_with_one_point_is_not_exported():
|
||||
store.aggregate("fn", CalSample(weight_n=1.0, raw_mean=10.0, n=10))
|
||||
with pytest.raises(ValueError):
|
||||
build_force_calibration_v2(store)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- one preparation
|
||||
|
||||
|
||||
def _session_with_duplicates_and_validation():
|
||||
recorder = CalibrationRecorder()
|
||||
points = [
|
||||
_point(recorder, 1.0, 100.0),
|
||||
_point(recorder, 2.0, 200.0),
|
||||
# A deliberate repeat of load 2.0 at a different raw level.
|
||||
_point(recorder, 2.0, 240.0),
|
||||
_point(recorder, 4.0, 400.0),
|
||||
_point(recorder, 3.0, 290.0),
|
||||
]
|
||||
recorder.set_validation(points[4], True)
|
||||
return recorder, points
|
||||
|
||||
|
||||
def test_duplicate_loads_are_merged_before_the_model_is_built():
|
||||
_recorder, _points = _session_with_duplicates_and_validation()
|
||||
prepared = _recorder.prepared()
|
||||
assert prepared.samples["fn"], "no training samples"
|
||||
loads = [sample.weight_n for sample in prepared.samples["fn"]]
|
||||
assert loads == sorted(set(loads)), f"duplicate loads were not merged: {loads}"
|
||||
assert len(loads) == 3
|
||||
merged = next(sample for sample in prepared.samples["fn"] if sample.weight_n == 2.0)
|
||||
assert merged.n == 16, "the repeat's samples were not combined"
|
||||
assert merged.raw_mean == pytest.approx((200.0 + 240.0) / 2.0)
|
||||
assert prepared.points_in_fit("fn") == 3
|
||||
|
||||
|
||||
def test_the_exported_file_predicts_exactly_what_the_panel_shows():
|
||||
"""Read the written document back and compare it with the diagnostics.
|
||||
|
||||
The panel's numbers must describe the file that is actually written, so the
|
||||
test loads the file with the SDK (not with the prepared object) and predicts
|
||||
every training and validation point from it.
|
||||
"""
|
||||
import orisys.calibration as sdk_calibration
|
||||
|
||||
recorder, points = _session_with_duplicates_and_validation()
|
||||
prepared = recorder.prepared(meta={"sensor_type": "box", "sn": "SN9"})
|
||||
assert prepared.ok
|
||||
|
||||
document = prepared.to_document()
|
||||
path = _root() / "force_calibration.json"
|
||||
write_force_calibration(document, path)
|
||||
from_file = sdk_calibration.load(path, strict=True)
|
||||
|
||||
for point in points:
|
||||
raw = float(point.loaded_stats().mean)
|
||||
predicted = float(from_file.get("fn").convert(raw))
|
||||
value = prepared.predict("fn", raw)
|
||||
assert predicted == pytest.approx(value, abs=1e-9)
|
||||
if point.is_validation:
|
||||
assert prepared.validation_errors[id(point)] == pytest.approx(
|
||||
abs(predicted - point.reference_load_n), abs=1e-9
|
||||
)
|
||||
else:
|
||||
assert prepared.residuals[id(point)] == pytest.approx(
|
||||
predicted - point.reference_load_n, abs=1e-9
|
||||
)
|
||||
|
||||
# The validation point is measured, never built in.
|
||||
assert prepared.validation_points("fn") == 1
|
||||
assert prepared.max_validation_error("fn") > 0.0
|
||||
exported_raws = sorted(point[0] for point in document["channels"]["normal"]["points"])
|
||||
assert exported_raws == [100.0, 220.0, 400.0], exported_raws
|
||||
|
||||
|
||||
def test_the_recorder_fit_and_the_preparation_agree():
|
||||
recorder, _points = _session_with_duplicates_and_validation()
|
||||
prepared = recorder.prepared()
|
||||
fit = recorder.fit("fn", prepared=prepared)
|
||||
assert fit.n_points == prepared.points_in_fit("fn")
|
||||
assert fit.breakpoints == prepared.breakpoints("fn")
|
||||
assert fit.max_residual_n == pytest.approx(prepared.max_residual("fn"))
|
||||
assert fit.validation_error_n == pytest.approx(prepared.max_validation_error("fn"))
|
||||
# ...and a fit that prepares on its own gets the same numbers.
|
||||
assert recorder.fit("fn").as_dict() == fit.as_dict()
|
||||
|
||||
@@ -31,15 +31,15 @@ def _root() -> Path:
|
||||
return SCRATCH
|
||||
|
||||
|
||||
def _recorded_point(index_offset: float = 0.0):
|
||||
def _recorded_point(load: float = 2.0, offset: float = 0.0):
|
||||
recorder = CalibrationRecorder()
|
||||
recorder.start(channel="fn", reference_load_n=2.0, duration_s=0.05)
|
||||
recorder.start(channel="fn", reference_load_n=load, duration_s=0.05)
|
||||
for index in range(6):
|
||||
recorder.feed(
|
||||
SessionSample(
|
||||
index,
|
||||
index * 1.0 / 30.0,
|
||||
fn_raw=float(index) + index_offset,
|
||||
fn_raw=float(index) + offset,
|
||||
fx_raw=1.0,
|
||||
fy_raw=2.0,
|
||||
fn_n=float(index) * 10.0,
|
||||
@@ -126,9 +126,9 @@ def test_a_newer_format_is_loaded_with_a_warning():
|
||||
|
||||
def test_the_recorder_can_reopen_a_saved_session():
|
||||
points = [
|
||||
_recorded_point(),
|
||||
_recorded_point(5.0),
|
||||
_recorded_point(9.0),
|
||||
_recorded_point(1.0),
|
||||
_recorded_point(2.0, 5.0),
|
||||
_recorded_point(3.0, 9.0),
|
||||
]
|
||||
path = write_calibration_session(_root() / "session.json", points)
|
||||
restored = read_calibration_session(path).points
|
||||
@@ -141,3 +141,76 @@ def test_the_recorder_can_reopen_a_saved_session():
|
||||
fit = recorder.fit("fn")
|
||||
assert fit.validation_points == 1
|
||||
assert fit.n_points == 2
|
||||
|
||||
|
||||
def test_reopened_points_at_one_load_are_one_calibrated_load():
|
||||
"""Three points at the same reference load are one load, not a curve.
|
||||
|
||||
They are merged (n-weighted) before the build, so a session that only ever
|
||||
sampled one load cannot produce a model — which is the honest answer.
|
||||
"""
|
||||
recorder = CalibrationRecorder()
|
||||
recorder.load_points(
|
||||
[_recorded_point(2.0), _recorded_point(2.0, 5.0), _recorded_point(2.0, 9.0)],
|
||||
replace=True,
|
||||
)
|
||||
prepared = recorder.prepared()
|
||||
assert len(prepared.training) == 3
|
||||
assert len(prepared.samples["fn"]) == 1
|
||||
assert prepared.samples["fn"][0].n == 15, "the valid counts were not summed"
|
||||
assert prepared.model is None, "a single load produced a model"
|
||||
assert recorder.fit("fn").n_points == 1
|
||||
|
||||
|
||||
# ------------------------------------------------------- reopened diagnostics
|
||||
|
||||
|
||||
def test_a_reopened_record_keeps_its_device_identity():
|
||||
"""Device A's record stays A's, whatever is connected now."""
|
||||
points = [_recorded_point(1.0), _recorded_point(2.0, 5.0)]
|
||||
path = write_calibration_session(
|
||||
_root() / "session.json",
|
||||
points,
|
||||
meta={
|
||||
"sensor_type": "box",
|
||||
"sn": "SN-A",
|
||||
"config_ref": "D:/lab/deviceA/sensor.json",
|
||||
},
|
||||
)
|
||||
record = read_calibration_session(path)
|
||||
identity = record.identity()
|
||||
assert identity["sensor_type"] == "box"
|
||||
assert identity["sn"] == "SN-A"
|
||||
assert identity["config_ref"] == "D:/lab/deviceA/sensor.json"
|
||||
assert identity["created_at"] == record.created_at
|
||||
assert "SN-A" in record.describe()
|
||||
|
||||
|
||||
def test_the_three_axis_trace_is_rebuilt_from_the_samples():
|
||||
"""A record whose trace was never written must still be inspectable."""
|
||||
import json
|
||||
|
||||
point = _recorded_point(1.0)
|
||||
document = build_session_document([point])
|
||||
# Simulate a document that only kept the audit series (or an older writer
|
||||
# that stored a truncated trace).
|
||||
document["points"][0]["trace"] = {}
|
||||
document["points"][0]["duration_s"] = 0.0
|
||||
path = _root() / "session.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
|
||||
restored = read_calibration_session(path).points[0]
|
||||
assert set(restored.trace) == {"fn", "fx", "fy"}, restored.trace
|
||||
assert len(restored.trace["fn"]) == 5 # the valid samples
|
||||
assert restored.trace_dt_s == pytest.approx(1.0 / 30.0, abs=1e-6)
|
||||
assert restored.duration_s == pytest.approx(5.0 / 30.0, abs=1e-6)
|
||||
|
||||
|
||||
def test_a_stored_trace_is_not_overwritten_on_reopen():
|
||||
point = _recorded_point(1.0)
|
||||
document = build_session_document([point])
|
||||
document["points"][0]["trace"] = {"fn": [1.0, 2.0]}
|
||||
path = _root() / "session.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
restored = read_calibration_session(path).points[0]
|
||||
assert restored.trace["fn"] == [1.0, 2.0]
|
||||
|
||||
@@ -16,7 +16,12 @@ from unittest.mock import patch
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.record.depth import DepthRecomputer
|
||||
from orisys_app_shared.record.depth import (
|
||||
STATE_APPROXIMATE,
|
||||
STATE_EXACT,
|
||||
STATE_PREROLL,
|
||||
DepthRecomputer,
|
||||
)
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
@@ -36,6 +41,12 @@ class _CountingSensor:
|
||||
_CountingSensor.opened.append({"args": args, "kwargs": kwargs})
|
||||
self.n = 0
|
||||
self.chain_calls = 0
|
||||
self.initial_img = "reference-frame"
|
||||
# The real Sensor opens its source in the constructor, so the first
|
||||
# source is part of its history too.
|
||||
self.vid_src = str(args[0]) if args else ""
|
||||
self.sources: list[str] = [self.vid_src] if self.vid_src else []
|
||||
self.resets = 0
|
||||
self.info = SimpleNamespace(DEPTH=1)
|
||||
|
||||
def get_img(self):
|
||||
@@ -48,9 +59,17 @@ class _CountingSensor:
|
||||
def compute_contact(self):
|
||||
self.chain_calls += 1
|
||||
|
||||
def reset(self):
|
||||
self.resets += 1
|
||||
|
||||
def read_info(self, _channel):
|
||||
return np.array([self.n])
|
||||
|
||||
def _open_video_source(self, _override, _box):
|
||||
self.sources.append(str(self.vid_src))
|
||||
# A real reopen restarts the file from its first frame.
|
||||
self.n = 0
|
||||
|
||||
def disconnect(self):
|
||||
return None
|
||||
|
||||
@@ -116,13 +135,18 @@ def test_stepping_continues_from_the_current_position():
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_stepping_backwards_reopens_the_segment():
|
||||
def test_stepping_backwards_reopens_the_source_but_keeps_the_processor():
|
||||
_CountingSensor.opened.clear()
|
||||
computer = DepthRecomputer(_reader(_session("/cfg/sensor.json")))
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
assert computer._depth_from_segment(Path("raw_0001.mp4"), 5).tolist() == [5]
|
||||
# Reading an earlier frame restarts the file; the processor object (and
|
||||
# therefore the tracker) is not thrown away.
|
||||
assert computer._depth_from_segment(Path("raw_0001.mp4"), 2).tolist() == [2]
|
||||
assert len(_CountingSensor.opened) == 2, "seeking backwards needs a fresh source"
|
||||
sensor = computer._sensor._sensor
|
||||
assert len(_CountingSensor.opened) == 1, "the processor was recreated"
|
||||
assert len(sensor.sources) == 2
|
||||
assert all(source.endswith("raw_0001.mp4") for source in sensor.sources)
|
||||
computer.close()
|
||||
|
||||
|
||||
@@ -165,11 +189,44 @@ def test_the_plan_starts_at_the_last_reset_in_the_same_segment():
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[8])
|
||||
assert plan.stop_at == 9
|
||||
assert plan.anchor == "reset"
|
||||
assert plan.state == STATE_EXACT
|
||||
assert plan.reset_frame_id == 6
|
||||
assert len(plan.steps) == 1
|
||||
step = plan.steps[0]
|
||||
assert step.source == reader.directory / "raw_0001.mp4"
|
||||
assert step.to_frame == 9
|
||||
assert step.reset_after == 7, "the reset boundary is not re-applied by the plan"
|
||||
assert plan.frames_to_read == 9
|
||||
assert plan.preroll is None
|
||||
assert plan.warnings == ()
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_reset_that_is_not_in_the_video_is_not_a_recovered_state():
|
||||
"""Finding the event is not the same as being able to start there."""
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1,
|
||||
segment_frame=index + 1,
|
||||
in_video=index != 6, # the reset frame never reached the video
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(10)
|
||||
]
|
||||
reader = _reader(_session("/cfg/sensor.json"), frames=frames)
|
||||
reader.last_reset_frame = lambda frame_id: 6
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[8])
|
||||
assert plan.anchor != "reset"
|
||||
assert plan.state == STATE_APPROXIMATE
|
||||
assert "reset_frame_not_in_video" in plan.warnings
|
||||
assert "state_approximate" in plan.warnings
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_the_plan_uses_the_pre_roll_for_the_start_of_a_recording():
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
@@ -195,6 +252,8 @@ def test_the_plan_uses_the_pre_roll_for_the_start_of_a_recording():
|
||||
plan = computer.plan_for(frames[3])
|
||||
assert plan.preroll is not None
|
||||
assert plan.preroll_frames == 30
|
||||
assert plan.anchor == "preroll"
|
||||
assert plan.state == STATE_PREROLL
|
||||
assert plan.warnings == ()
|
||||
computer.close()
|
||||
|
||||
@@ -215,10 +274,127 @@ def test_a_segment_without_history_is_flagged_approximate():
|
||||
reader = _reader(_session("/cfg/sensor.json"), frames=frames)
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[2])
|
||||
assert plan.state == STATE_APPROXIMATE
|
||||
assert plan.anchor == "segment_start"
|
||||
assert "state_approximate" in plan.warnings
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_reset_in_an_earlier_segment_is_carried_across_by_swapping_the_source():
|
||||
"""The processor survives the swap, so the state *can* cross a segment."""
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1 if index < 3 else 2,
|
||||
segment_frame=index + 1 if index < 3 else index - 2,
|
||||
in_video=True,
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(6)
|
||||
]
|
||||
reader = _reader(
|
||||
_session("/cfg/sensor.json"),
|
||||
segments=("raw_0001.mp4", "raw_0002.mp4"),
|
||||
frames=frames,
|
||||
)
|
||||
reader.last_reset_frame = lambda frame_id: 1
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[5])
|
||||
assert plan.anchor == "reset"
|
||||
assert plan.state == STATE_EXACT
|
||||
assert "state_across_segment" not in plan.warnings
|
||||
assert [step.source.name for step in plan.steps] == ["raw_0001.mp4", "raw_0002.mp4"]
|
||||
assert plan.steps[0].reset_after == 2
|
||||
assert plan.steps[1].to_frame == 3
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_replay_that_would_read_too_far_back_is_reported_not_attempted():
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1,
|
||||
segment_frame=index + 1,
|
||||
in_video=True,
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(20)
|
||||
]
|
||||
reader = _reader(_session("/cfg/sensor.json"), frames=frames)
|
||||
reader.last_reset_frame = lambda frame_id: 1
|
||||
computer = DepthRecomputer(reader)
|
||||
with patch("orisys_app_shared.record.depth.MAX_REPLAY_FRAMES", 5):
|
||||
plan = computer.plan_for(frames[10])
|
||||
assert "state_too_far_back" in plan.warnings
|
||||
assert plan.state == STATE_APPROXIMATE
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_the_same_frame_is_answered_from_the_cache():
|
||||
"""A second request for the frame just read is a cache hit, not None."""
|
||||
_CountingSensor.opened.clear()
|
||||
computer = DepthRecomputer(_reader(_session("/cfg/sensor.json")))
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
first = computer._depth_from_segment(Path("raw_0001.mp4"), 3)
|
||||
second = computer._depth_from_segment(Path("raw_0001.mp4"), 3)
|
||||
sensor = computer._sensor._sensor
|
||||
assert first.tolist() == [3]
|
||||
assert second is not None and second.tolist() == [3]
|
||||
assert sensor.n == 3, "the cached answer re-read the stream"
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_the_processor_survives_a_pre_roll_to_segment_switch():
|
||||
"""Only the video source is replaced; the tracker state must survive."""
|
||||
_CountingSensor.opened.clear()
|
||||
reader = _reader(_session("/cfg/sensor.json"))
|
||||
preroll = reader.directory / "preroll_0001.mp4"
|
||||
preroll.write_bytes(b"")
|
||||
computer = DepthRecomputer(reader)
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
computer._feed(preroll, 2)
|
||||
sensor = computer._sensor._sensor
|
||||
assert sensor.n == 2, "the pre-roll frames were not fed"
|
||||
# Feed the segment: same processor, new source, position restarted.
|
||||
computer._feed(reader.directory / "raw_0001.mp4", 1)
|
||||
assert len(_CountingSensor.opened) == 1, "the processor was recreated"
|
||||
assert computer._sensor._sensor is sensor
|
||||
assert len(sensor.sources) == 2
|
||||
assert sensor.sources[0].endswith("preroll_0001.mp4")
|
||||
assert sensor.sources[1].endswith("raw_0001.mp4")
|
||||
assert sensor.initial_img == "reference-frame", "the tracker reference was reset"
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_reset_boundary_is_re_applied_during_the_replay():
|
||||
"""The live zeroing cleared the tracker; the replay must do the same."""
|
||||
_CountingSensor.opened.clear()
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1,
|
||||
segment_frame=index + 1,
|
||||
in_video=True,
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(10)
|
||||
]
|
||||
reader = _reader_with_config(frames)
|
||||
reader.last_reset_frame = lambda frame_id: 6
|
||||
reader.frame_at = lambda _t: frames[8]
|
||||
computer = DepthRecomputer(reader)
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
result = computer.depth_at(frames[8].t_rel_s)
|
||||
sensor = computer._sensor._sensor
|
||||
assert result.ok, result.error
|
||||
assert sensor.resets == 1, "the reset boundary was not re-applied"
|
||||
assert result.warnings == (), result.warnings
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_dropped_frame_is_reported_not_substituted():
|
||||
"""The index marks the hole; replay must say so instead of showing a neighbour."""
|
||||
frames = [
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Display unit selection and the per-axis drawing rule.
|
||||
|
||||
Two defects are covered here:
|
||||
|
||||
* the live charts chose their unit from the device-level ``force_calibrated``
|
||||
flag, so a device with only its normal-force axis calibrated either showed
|
||||
every axis in raw counts or showed raw counts inside newton panels;
|
||||
* the drawing rule asked for **all three** axes to be finite, so a valid FN
|
||||
disappeared from the chart whenever FX/FY had no calibrated value.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
from orisys_app_shared.force import select_display_forces
|
||||
from orisys_app_shared.force.session import SessionSample, SessionStore
|
||||
from orisys_app_shared.record.sample import to_record_sample
|
||||
|
||||
NAN = float("nan")
|
||||
|
||||
|
||||
def _packet(**overrides) -> SdkPacket:
|
||||
frame = np.zeros((4, 4, 3), np.uint8)
|
||||
kwargs = dict(
|
||||
raw_frame=frame,
|
||||
processed_frame=frame,
|
||||
timestamp=1000.0,
|
||||
fps=30.0,
|
||||
process_ms=1.0,
|
||||
normal_force=1.5,
|
||||
shear_x=0.25,
|
||||
shear_y=-0.25,
|
||||
normal_force_raw=123.0,
|
||||
shear_x_raw=2.0,
|
||||
shear_y_raw=-2.0,
|
||||
force_calibrated=True,
|
||||
calibrated_axes=("fn", "fx", "fy"),
|
||||
frame_id=0,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return SdkPacket(**kwargs)
|
||||
|
||||
|
||||
def test_newton_mode_marks_uncalibrated_axes_unavailable():
|
||||
sample = to_record_sample(_packet(calibrated_axes=("fn",)))
|
||||
display = select_display_forces(units_pref="newton", sample=sample)
|
||||
assert display.unit == "N"
|
||||
assert display.values[0] == pytest.approx(1.5)
|
||||
assert math.isnan(display.values[1]) and math.isnan(display.values[2])
|
||||
assert display.available == (True, False, False)
|
||||
assert display.unavailable_axes() == ("fx", "fy")
|
||||
assert not display.using_raw_fallback
|
||||
|
||||
|
||||
def test_raw_mode_shows_the_raw_counts_for_every_axis():
|
||||
sample = to_record_sample(_packet(calibrated_axes=("fn",)))
|
||||
display = select_display_forces(units_pref="raw", sample=sample)
|
||||
assert display.unit == "raw"
|
||||
assert display.values == (123.0, 2.0, -2.0)
|
||||
assert display.available == (True, True, True)
|
||||
assert not display.using_raw_fallback
|
||||
|
||||
|
||||
def test_an_uncalibrated_device_falls_back_to_raw_and_says_so():
|
||||
sample = to_record_sample(
|
||||
_packet(force_calibrated=False, calibrated_axes=())
|
||||
)
|
||||
display = select_display_forces(units_pref="newton", sample=sample)
|
||||
assert display.unit == "raw"
|
||||
assert display.using_raw_fallback, "the fallback must be announceable"
|
||||
assert display.requested_unit == "N"
|
||||
assert display.values == (123.0, 2.0, -2.0)
|
||||
|
||||
|
||||
def test_a_failed_readout_has_no_available_axis():
|
||||
sample = to_record_sample(
|
||||
_packet(valid=False, invalid_reason="compute failed", stale_forces=True)
|
||||
)
|
||||
for unit in ("N", "raw"):
|
||||
display = select_display_forces(units_pref=unit, sample=sample)
|
||||
assert display.available == (False, False, False)
|
||||
assert all(math.isnan(value) for value in display.values)
|
||||
|
||||
|
||||
def _store_with_one_calibrated_axis() -> SessionStore:
|
||||
store = SessionStore()
|
||||
for index in range(5):
|
||||
store.append(
|
||||
SessionSample(
|
||||
index,
|
||||
index / 30.0,
|
||||
fn_raw=100.0 + index,
|
||||
fx_raw=2.0,
|
||||
fy_raw=-2.0,
|
||||
fn_n=1.0 + index * 0.1,
|
||||
fx_n=NAN,
|
||||
fy_n=NAN,
|
||||
valid=True,
|
||||
calibrated_axes=("fn",),
|
||||
)
|
||||
)
|
||||
return store
|
||||
|
||||
|
||||
def test_a_valid_axis_is_drawn_even_when_the_others_have_no_value():
|
||||
"""The all-axes rule hid a good FN because FX/FY were uncalibrated."""
|
||||
slice_ = _store_with_one_calibrated_axis().data()
|
||||
times, values, mask = slice_.plottable("N")
|
||||
assert mask.shape == (5, 3)
|
||||
assert mask[:, 0].all(), "the calibrated axis was hidden"
|
||||
assert not mask[:, 1].any()
|
||||
assert not mask[:, 2].any()
|
||||
|
||||
fn_times, fn_values = slice_.plottable_axis("N", 0)
|
||||
assert fn_times.size == 5 == fn_values.size
|
||||
assert float(fn_values.max()) == pytest.approx(1.4)
|
||||
fx_times, _fx_values = slice_.plottable_axis("N", 1)
|
||||
assert fx_times.size == 0
|
||||
|
||||
|
||||
def test_the_raw_triplet_is_still_complete_for_the_same_samples():
|
||||
slice_ = _store_with_one_calibrated_axis().data()
|
||||
for index in range(3):
|
||||
times, values = slice_.plottable_axis("raw", index)
|
||||
assert times.size == 5
|
||||
assert np.isfinite(values).all()
|
||||
|
||||
|
||||
def test_an_invalid_sample_is_excluded_from_every_axis():
|
||||
store = _store_with_one_calibrated_axis()
|
||||
store.append(SessionSample(9, 9.0, fn_n=NAN, valid=False))
|
||||
slice_ = store.data()
|
||||
_times, _values, mask = slice_.plottable("N")
|
||||
assert not mask[-1].any(), "an invalid sample reached a drawing path"
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Long-history browsing: interval reads, envelopes and one curve interface.
|
||||
|
||||
The replay used to load every row of a recording before drawing anything, so a
|
||||
multi-hour session could not be shown at all. These tests pin the two things that
|
||||
make it possible — an interval read bounded by the interval, and a whole-session
|
||||
envelope built while streaming the file once — and that the charts can read from
|
||||
either a session store or a recording through the same interface.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
from orisys_app_shared.force.session import SessionSample, SessionStore
|
||||
from orisys_app_shared.force.source import RecordingCurveSource, StoreCurveSource
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig, RecordingReader
|
||||
from orisys_app_shared.record.index import RecordingIndex
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch_history"
|
||||
|
||||
ROWS = 12_000
|
||||
SPIKE_ROW = 9_000
|
||||
SPIKE_VALUE = 999.0
|
||||
|
||||
|
||||
def _root() -> Path:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
return SCRATCH
|
||||
|
||||
|
||||
def _packet(index: int) -> SdkPacket:
|
||||
frame = np.zeros((4, 4, 3), np.uint8)
|
||||
fx = SPIKE_VALUE if index == SPIKE_ROW else 0.5
|
||||
return SdkPacket(
|
||||
raw_frame=frame,
|
||||
processed_frame=frame,
|
||||
timestamp=1000.0 + index / 30.0,
|
||||
mono_t=index / 30.0,
|
||||
fps=30.0,
|
||||
process_ms=1.0,
|
||||
normal_force=1.0,
|
||||
shear_x=fx,
|
||||
shear_y=-0.5,
|
||||
normal_force_raw=10.0 + (index % 7) * 0.01,
|
||||
shear_x_raw=fx,
|
||||
shear_y_raw=-0.5,
|
||||
force_unit="N",
|
||||
force_calibrated=True,
|
||||
frame_id=index,
|
||||
)
|
||||
|
||||
|
||||
def _recording() -> Path:
|
||||
"""A recording of ``ROWS`` rows with no drops: the writer is allowed to keep up."""
|
||||
import time
|
||||
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
for index in range(ROWS):
|
||||
recorder.on_packet(_packet(index))
|
||||
if index % 256 == 0:
|
||||
# Feeding 12000 rows in a tight loop would (correctly) overflow the
|
||||
# ordered stream and be recorded as gaps; this test is about history
|
||||
# browsing, so let the writer drain.
|
||||
deadline = time.monotonic() + 5.0
|
||||
while recorder._queue.qsize() > 512 and time.monotonic() < deadline:
|
||||
time.sleep(0.001)
|
||||
report = recorder.stop()
|
||||
assert report.stats.dropped_data == 0, report.problems
|
||||
assert report.stats.frames == ROWS
|
||||
return Path(directory)
|
||||
|
||||
|
||||
def test_the_index_reads_only_the_interval_it_is_asked_for():
|
||||
directory = _recording()
|
||||
index = RecordingIndex(directory)
|
||||
index.build()
|
||||
assert index.rows == ROWS
|
||||
assert index.duration_s() == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
|
||||
|
||||
# The whole range keeps every row, including the very first one.
|
||||
whole = index.read_rows(0, ROWS)
|
||||
assert len(whole) == ROWS
|
||||
assert whole.frame_ids[0] == 0
|
||||
assert whole.frame_ids[-1] == ROWS - 1
|
||||
|
||||
start, stop = index.row_range(300.0, 300.5)
|
||||
# One row before the interval is included on purpose: a cursor exactly on a
|
||||
# sample still wants the row it follows.
|
||||
assert 15 <= stop - start <= 20, (start, stop)
|
||||
window = index.read_rows(start, stop)
|
||||
assert 15 <= len(window) <= 20
|
||||
assert window.times[0] >= 300.0 - 1 / 30.0
|
||||
assert window.times[-1] <= 300.5
|
||||
|
||||
|
||||
def test_a_large_recording_is_loaded_as_a_marked_decimated_view():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
info = reader.info()
|
||||
assert info.decimated, "a 12000-row recording was loaded in full"
|
||||
assert info.total_rows == ROWS
|
||||
assert len(info.times) <= 500
|
||||
# The duration comes from the frame index, never from the decimated view.
|
||||
assert info.duration_s == pytest.approx((ROWS - 1) / 30.0, abs=0.05)
|
||||
assert len(info.frames) == ROWS
|
||||
|
||||
|
||||
def test_a_small_recording_is_not_decimated():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=50_000)
|
||||
info = reader.info()
|
||||
assert not info.decimated
|
||||
assert len(info.times) == ROWS
|
||||
assert info.total_rows == ROWS
|
||||
|
||||
|
||||
def test_an_interval_read_sees_every_row_including_a_single_peak():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
spike_t = SPIKE_ROW / 30.0
|
||||
window = reader.window(spike_t - 0.05, spike_t + 0.05)
|
||||
assert window.times.size >= 3
|
||||
assert float(np.nanmax(window.value_for_unit("N")[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
# ...and it is the *exact* interval, not the decimated whole-session view.
|
||||
assert not window.decimated
|
||||
assert window.times[0] >= spike_t - 0.1
|
||||
|
||||
|
||||
def test_a_decimated_window_still_keeps_the_peak():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
window = reader.window(0.0, (ROWS - 1) / 30.0, max_points=300, unit="N")
|
||||
assert window.decimated
|
||||
assert len(window) <= 600
|
||||
assert float(np.nanmax(window.value_for_unit("N")[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
|
||||
|
||||
def test_the_envelope_covers_the_whole_session_in_few_points():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
overview = reader.overview(unit="N", buckets=200)
|
||||
assert 0 < len(overview) <= 200
|
||||
assert not np.isnan(overview.highs[:, 1]).all()
|
||||
# The spike is inside a bucket the strip would otherwise skip.
|
||||
assert float(np.nanmax(overview.highs[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
assert float(overview.counts.sum()) > 0
|
||||
# Cached: asking again does not re-stream the file.
|
||||
assert reader.overview(unit="N", buckets=200) is overview
|
||||
|
||||
|
||||
def test_the_two_sources_answer_the_same_questions():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=ROWS + 1)
|
||||
recording = RecordingCurveSource(reader)
|
||||
|
||||
store = SessionStore()
|
||||
for index in range(ROWS):
|
||||
sample = SessionSample(
|
||||
index,
|
||||
index / 30.0,
|
||||
fn_raw=10.0 + (index % 7) * 0.01,
|
||||
fx_raw=SPIKE_VALUE if index == SPIKE_ROW else 0.5,
|
||||
fy_raw=-0.5,
|
||||
fn_n=1.0,
|
||||
fx_n=SPIKE_VALUE if index == SPIKE_ROW else 0.5,
|
||||
fy_n=-0.5,
|
||||
)
|
||||
store.append(sample)
|
||||
memory = StoreCurveSource(store)
|
||||
|
||||
for source in (recording, memory):
|
||||
first, last = source.extent()
|
||||
assert first == pytest.approx(0.0, abs=1e-6)
|
||||
assert last == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
|
||||
assert source.duration_s() == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
|
||||
assert recording.kind == "disk" and memory.kind == "memory"
|
||||
|
||||
# The same interval, read from disk and from memory, agrees on the values.
|
||||
# (The disk read also includes the row just before the interval.)
|
||||
disk = recording.window(200.0, 210.0, unit="N")
|
||||
cached = memory.window(200.0, 210.0, unit="N")
|
||||
inside = (disk.times >= 200.0) & (disk.times <= 210.0)
|
||||
assert inside.sum() == len(cached) > 0
|
||||
assert np.allclose(disk.values[inside], cached.values, equal_nan=True)
|
||||
# The CSV keeps six decimals; the in-memory store keeps the raw float.
|
||||
assert np.allclose(disk.times[inside], cached.times, atol=1e-6)
|
||||
|
||||
# ...and so do the statistics of that interval (the disk read may include the
|
||||
# one row that precedes it).
|
||||
a = recording.statistics(200.0, 210.0, unit="N")
|
||||
b = memory.statistics(200.0, 210.0, unit="N")
|
||||
assert abs(a.samples - b.samples) <= 1
|
||||
assert a.peaks[1] == pytest.approx(b.peaks[1])
|
||||
assert a.means[1] == pytest.approx(b.means[1], abs=1e-6)
|
||||
|
||||
|
||||
def test_the_whole_session_can_be_read_without_holding_it_all():
|
||||
"""The point of the index: peak value with a bounded number of rows held."""
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
overview = reader.overview(unit="N", buckets=100)
|
||||
# What a whole-session strip needs is one band per bucket, not 12000 rows.
|
||||
assert len(overview) <= 100
|
||||
assert len(reader.info().times) <= 500
|
||||
assert reader.info().total_rows == ROWS
|
||||
# A single sample can still be read exactly, on demand.
|
||||
spike_t = SPIKE_ROW / 30.0
|
||||
exact = reader.window(spike_t, spike_t + 1 / 30.0)
|
||||
assert 1 <= len(exact) <= 3, len(exact)
|
||||
assert float(np.nanmax(exact.newtons[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
@@ -0,0 +1,105 @@
|
||||
"""The module boundaries the responsibility split is supposed to create.
|
||||
|
||||
A pure move is only worth anything if the dependency directions actually change:
|
||||
the finger worker must not need the box worker to describe a packet, the session
|
||||
file format must not need the sampling state machine, and the models must not
|
||||
import either. These assertions are cheap and they fail loudly if someone
|
||||
re-introduces the cycle.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
SHARED = Path(__file__).resolve().parents[1] / "src"
|
||||
ACQUISITION = SHARED / "orisys_app_shared" / "acquisition"
|
||||
FORCE = SHARED / "orisys_app_shared" / "force"
|
||||
RECORD = SHARED / "orisys_app_shared" / "record"
|
||||
|
||||
|
||||
def _imports(path: Path) -> set[str]:
|
||||
"""Every module name imported by ``path`` (``from x import y`` counts as x)."""
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
found: set[str] = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
found.add(alias.name)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
if node.module:
|
||||
found.add(node.module)
|
||||
# ``from . import models`` also names the submodule.
|
||||
for alias in node.names:
|
||||
found.add(f"{node.module}.{alias.name}" if node.module else alias.name)
|
||||
return found
|
||||
|
||||
|
||||
def _imports_any(path: Path, needles) -> bool:
|
||||
found = _imports(path)
|
||||
for needle in needles:
|
||||
if any(entry == needle or entry.endswith(f".{needle}") for entry in found):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def test_the_packet_model_does_not_import_a_worker():
|
||||
assert not _imports_any(
|
||||
ACQUISITION / "models.py", {"sdk_worker", "finger_worker"}
|
||||
), "acquisition.models depends on a worker"
|
||||
|
||||
|
||||
def test_the_finger_worker_takes_the_packet_from_models():
|
||||
"""It used to import the box worker just to describe a frame."""
|
||||
source = (ACQUISITION / "finger_worker.py").read_text(encoding="utf-8")
|
||||
tree = ast.parse(source)
|
||||
from_sdk_worker: set[str] = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and (node.module or "").endswith("sdk_worker"):
|
||||
from_sdk_worker.update(alias.name for alias in node.names)
|
||||
assert "SdkPacket" not in from_sdk_worker, "the finger worker still borrows the packet"
|
||||
assert "FixedCameraMode" not in from_sdk_worker
|
||||
assert "load_fixed_mode" not in from_sdk_worker
|
||||
# ...and the helpers it genuinely shares still come from there.
|
||||
assert "process_chain" in from_sdk_worker
|
||||
|
||||
|
||||
def test_the_calibration_models_do_not_import_the_sampler_or_the_session_file():
|
||||
for path in (FORCE / "models.py",):
|
||||
assert not _imports_any(
|
||||
path, {"calibration_recorder", "calibration_session", "session"}
|
||||
), f"{path.name} depends on a higher layer"
|
||||
|
||||
|
||||
def test_the_session_format_does_not_import_the_sampling_state_machine():
|
||||
assert not _imports_any(
|
||||
FORCE / "calibration_session.py", {"calibration_recorder"}
|
||||
), "the session file format still depends on the sampler"
|
||||
|
||||
|
||||
def test_the_moved_models_are_the_same_objects_everywhere():
|
||||
from orisys_app_shared.acquisition import finger_worker, models, sdk_worker
|
||||
from orisys_app_shared.force import calibration_recorder, calibration_session, models as force_models
|
||||
|
||||
assert sdk_worker.SdkPacket is models.SdkPacket is finger_worker.SdkPacket
|
||||
assert sdk_worker.FixedCameraMode is models.FixedCameraMode
|
||||
assert calibration_recorder.CalibrationPoint is force_models.CalibrationPoint
|
||||
assert calibration_recorder.AxisStats is force_models.AxisStats
|
||||
assert calibration_session.SAMPLE_FIELDS is force_models.SAMPLE_FIELDS
|
||||
# The recorder still offers the trace helpers it used to own.
|
||||
assert callable(calibration_recorder.trace_from)
|
||||
assert callable(calibration_recorder.trace_dt)
|
||||
|
||||
|
||||
def test_the_recording_pieces_are_separate_modules():
|
||||
from orisys_app_shared.record import depth, experiment
|
||||
from orisys_app_shared.record.depth_backends import BoxReplay, FingerReplay
|
||||
from orisys_app_shared.record.video_writer import VideoSegments
|
||||
|
||||
# The depth replay no longer defines its backends inline...
|
||||
assert not hasattr(depth, "BoxReplay")
|
||||
assert not hasattr(depth, "_BoxReplay")
|
||||
# ...and the writer delegates video encoding to its own component.
|
||||
assert not hasattr(experiment, "_open_segment")
|
||||
assert callable(VideoSegments.write) and callable(VideoSegments.close)
|
||||
assert BoxReplay.__name__ == "BoxReplay" and FingerReplay.__name__ == "FingerReplay"
|
||||
@@ -137,6 +137,78 @@ def test_reader_window_and_streaming_agree(recording):
|
||||
assert streamed == 61, "streaming must see every row"
|
||||
|
||||
|
||||
def test_an_empty_recording_directory_is_readable():
|
||||
"""A folder with no CSVs must open (empty), not raise an unpack error."""
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
reader = RecordingReader(SCRATCH)
|
||||
info = reader.info()
|
||||
assert info.frames == []
|
||||
assert info.times.size == 0
|
||||
assert info.duration_s == 0.0
|
||||
assert reader.frame_at(1.0) is None
|
||||
assert reader.replay_warnings(sdk_version="0.5.1")
|
||||
finally:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
|
||||
|
||||
def test_a_new_recording_is_reported_as_time_ordered(recording):
|
||||
info = RecordingReader(recording).info()
|
||||
assert info.frames_out_of_order, "the writer no longer produces ordered rows"
|
||||
assert info.forces_out_of_order
|
||||
codes = RecordingReader(recording).replay_warnings(sdk_version="0.5.1")
|
||||
assert "out_of_order" not in codes
|
||||
|
||||
|
||||
def test_an_old_out_of_order_recording_is_re_sorted_and_reported():
|
||||
"""Rows written by the pre-fix writer are not in time order.
|
||||
|
||||
``frame_at`` binary-searches these times, so an unsorted file would answer
|
||||
with the wrong frame; the reader sorts and says so instead.
|
||||
"""
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
(SCRATCH / "forces.csv").write_text(
|
||||
"session_id,frame_id,t_rel_s,FN_raw,FX_raw,FY_raw,FN_N,FX_N,FY_N,valid,calibration_id\n"
|
||||
"1,0,0.0,1.0,0.1,0.1,2.0,0.1,0.1,1,h\n"
|
||||
"1,2,0.066,3.0,0.1,0.1,6.0,0.1,0.1,1,h\n" # written before frame 1
|
||||
"1,1,0.033,2.0,0.1,0.1,4.0,0.1,0.1,1,h\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(SCRATCH / "frames.csv").write_text(
|
||||
"frame_id,t_rel_s,acquired_at,segment,segment_frame,in_video,dropped\n"
|
||||
"0,0.0,2026-01-01T00:00:00.000Z,1,1,1,0\n"
|
||||
"2,0.066,2026-01-01T00:00:00.066Z,1,3,1,0\n"
|
||||
"1,0.033,2026-01-01T00:00:00.033Z,1,2,1,0\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(SCRATCH / "session.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"format_version": "1.1",
|
||||
"device": {"sensor_type": "box", "config_ref": "/cfg/sensor.json"},
|
||||
"versions": {"sdk": "0.5.1"},
|
||||
"depth": {"stored": False},
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
try:
|
||||
reader = RecordingReader(SCRATCH)
|
||||
info = reader.info()
|
||||
assert not info.frames_out_of_order
|
||||
assert not info.forces_out_of_order
|
||||
assert [frame.frame_id for frame in info.frames] == [0, 1, 2]
|
||||
assert info.times.tolist() == pytest.approx([0.0, 0.033, 0.066])
|
||||
# The binary search is only valid because the rows were sorted.
|
||||
assert reader.frame_at(0.05).frame_id == 1
|
||||
assert reader.frame_at(0.07).frame_id == 2
|
||||
assert "out_of_order" in reader.replay_warnings(sdk_version="0.5.1")
|
||||
finally:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
|
||||
|
||||
def test_depth_policy_says_recompute_and_warns_about_drift(recording):
|
||||
reader = RecordingReader(recording)
|
||||
|
||||
|
||||
@@ -10,11 +10,9 @@ from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import queue
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
@@ -23,7 +21,11 @@ import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig
|
||||
from orisys_app_shared.record.experiment import STATE_FAILED, STATE_STOPPED, _SessionWriter
|
||||
from orisys_app_shared.record.experiment import (
|
||||
STATE_FAILED,
|
||||
STATE_STOPPED,
|
||||
_SessionWriter,
|
||||
)
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
@@ -189,21 +191,28 @@ def test_a_full_queue_keeps_the_rows_and_sheds_the_video():
|
||||
|
||||
|
||||
def test_a_lost_row_is_recorded_as_an_index_gap():
|
||||
"""A hole in the data must be visible, not silently bridged in replay."""
|
||||
"""A hole in the data must be visible, not silently bridged in replay.
|
||||
|
||||
The row is only given up when the ordered stream itself is full; the gap
|
||||
marker then carries the stream position, so it lands in the index where the
|
||||
hole actually is instead of after newer frames.
|
||||
"""
|
||||
original = _SessionWriter.write_sample
|
||||
calls = {"count": 0}
|
||||
|
||||
def slow(self, *args):
|
||||
calls["count"] += 1
|
||||
if calls["count"] <= 2:
|
||||
# Stall just long enough for the queue and the tiny backlog to fill.
|
||||
# Stall just long enough for the stream to pass the (tiny) cap.
|
||||
time.sleep(0.3)
|
||||
return original(self, *args)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False, max_queue=16))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder._row_backlog = deque(recorder._row_backlog, maxlen=2)
|
||||
with patch(
|
||||
"orisys_app_shared.record.experiment.ROW_BACKLOG", 2
|
||||
):
|
||||
for index in range(40):
|
||||
recorder.on_packet(_packet(index))
|
||||
report = recorder.stop()
|
||||
@@ -218,6 +227,57 @@ def test_a_lost_row_is_recorded_as_an_index_gap():
|
||||
assert report.stats.dropped_gaps > 0, "the loss was not marked in the index"
|
||||
|
||||
|
||||
def test_the_frame_index_stays_in_time_order_when_rows_are_lost():
|
||||
"""The old writer drained its backlog before the queue, so the index jumped."""
|
||||
original = _SessionWriter.write_sample
|
||||
calls = {"count": 0}
|
||||
|
||||
def slow(self, *args):
|
||||
calls["count"] += 1
|
||||
if calls["count"] <= 3:
|
||||
time.sleep(0.2)
|
||||
return original(self, *args)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=True, max_queue=4))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
with patch("orisys_app_shared.record.experiment.ROW_BACKLOG", 3):
|
||||
for index in range(60):
|
||||
recorder.on_packet(_packet(index))
|
||||
recorder.stop()
|
||||
|
||||
with (Path(directory) / "frames.csv").open(encoding="utf-8", newline="") as handle:
|
||||
frame_rows = list(csv.DictReader(handle))
|
||||
times = [float(row["t_rel_s"]) for row in frame_rows]
|
||||
assert times == sorted(times), "frames.csv is not in time order"
|
||||
ids = [int(row["frame_id"]) for row in frame_rows]
|
||||
assert ids == sorted(ids), "forces.csv/frames.csv disagree on order"
|
||||
with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
|
||||
force_rows = list(csv.DictReader(handle))
|
||||
force_times = [float(row["t_rel_s"]) for row in force_rows]
|
||||
assert force_times == sorted(force_times), "forces.csv is not in time order"
|
||||
|
||||
|
||||
def test_the_video_byte_budget_is_released_when_a_frame_is_shed():
|
||||
"""A shed frame must not leak its reservation for the rest of the session."""
|
||||
recorder = ExperimentRecorder(
|
||||
RecordConfig(video=True, max_queue=16, max_queue_bytes=4096)
|
||||
)
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
# Frames of 8x8x3 = 192 bytes, so the budget holds ~21 of them; without the
|
||||
# release the counter would climb past the budget and stay there.
|
||||
for index in range(200):
|
||||
recorder.on_packet(_packet(index))
|
||||
report = recorder.stop()
|
||||
assert report.stats.frames == 200, "rows were lost to the video budget"
|
||||
assert abs(recorder._queued_bytes) < 4096, (
|
||||
f"the byte budget leaked: {recorder._queued_bytes} bytes still reserved"
|
||||
)
|
||||
with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
|
||||
rows = list(csv.DictReader(handle))
|
||||
assert len(rows) == 200
|
||||
|
||||
|
||||
def test_session_json_names_the_pre_roll_when_one_was_written():
|
||||
frames = [np.full((8, 8, 3), index, np.uint8) for index in range(3)]
|
||||
recorder = ExperimentRecorder(RecordConfig(video=True, fps=5.0))
|
||||
@@ -241,6 +301,80 @@ def test_a_data_only_recording_says_why_it_has_no_pre_roll():
|
||||
assert any("pre-roll" in problem for problem in report.problems)
|
||||
|
||||
|
||||
def test_a_stalled_encoder_leaves_a_readable_recording():
|
||||
"""Order, duration, positioning and the byte counter after a stall.
|
||||
|
||||
Stall the writer, push past the ordered stream's cap, let it recover, then
|
||||
push again and read the directory back the way the replay does. The frames
|
||||
acquired *after* the recovery must all be there: an overflow is a temporary
|
||||
loss, not the end of the recording.
|
||||
"""
|
||||
from orisys_app_shared.record import RecordingReader
|
||||
|
||||
original = _SessionWriter.write_sample
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
calls = {"count": 0}
|
||||
|
||||
def slow(self, *args):
|
||||
calls["count"] += 1
|
||||
if calls["count"] == 2:
|
||||
started.set()
|
||||
release.wait(10.0)
|
||||
return original(self, *args)
|
||||
|
||||
def drained(recorder, timeout: float = 10.0) -> bool:
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if recorder._queue.qsize() == 0:
|
||||
return True
|
||||
time.sleep(0.01)
|
||||
return False
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=True, fps=30.0, max_queue=4))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
with patch("orisys_app_shared.record.experiment.ROW_BACKLOG", 3):
|
||||
for index in range(40):
|
||||
recorder.on_packet(_packet(index))
|
||||
assert started.wait(10.0), "the stall never happened"
|
||||
release.set()
|
||||
assert drained(recorder), "the writer never caught up"
|
||||
before = recorder.stats().frames
|
||||
for index in range(40, 100):
|
||||
recorder.on_packet(_packet(index))
|
||||
time.sleep(0.005)
|
||||
assert drained(recorder), "the recovered writer never caught up"
|
||||
report = recorder.stop()
|
||||
|
||||
assert not report.stats.failed, report.problems
|
||||
assert recorder.stats().frames > before, "recording did not continue after the stall"
|
||||
|
||||
reader = RecordingReader(directory)
|
||||
info = reader.info()
|
||||
times = info.times.tolist()
|
||||
ids = [int(value) for value in info.frame_ids]
|
||||
assert times == sorted(times), "forces.csv is not in time order"
|
||||
assert ids == sorted(ids)
|
||||
frame_times = [frame.t_rel_s for frame in info.frames]
|
||||
assert frame_times == sorted(frame_times), "frames.csv is not in time order"
|
||||
|
||||
# Everything acquired after the recovery is present.
|
||||
assert set(range(40, 100)).issubset(set(ids)), (
|
||||
f"frames lost after the encoder recovered: {sorted(set(range(40, 100)) - set(ids))}"
|
||||
)
|
||||
assert report.stats.duration_s == pytest.approx(99 / 30.0, abs=0.05)
|
||||
assert info.duration_s == pytest.approx(99 / 30.0, abs=0.05)
|
||||
|
||||
# Positioning: the frame at 2 s is the frame acquired at 2 s.
|
||||
frame = reader.frame_at(2.0)
|
||||
assert frame is not None and frame.frame_id == 60, frame
|
||||
assert frame.t_rel_s == pytest.approx(2.0, abs=0.05)
|
||||
|
||||
# The byte budget is back to zero once every frame has been consumed.
|
||||
assert recorder._queued_bytes == 0, recorder._queued_bytes
|
||||
|
||||
|
||||
def test_stopping_twice_returns_the_same_report():
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
recorder.start(root=_root(), session_id=1, metadata={})
|
||||
@@ -249,3 +383,146 @@ def test_stopping_twice_returns_the_same_report():
|
||||
second = recorder.stop()
|
||||
assert first.directory == second.directory
|
||||
assert second.ok
|
||||
|
||||
|
||||
# ------------------------------------------------------------ failure handling
|
||||
|
||||
|
||||
def test_a_directory_that_cannot_be_created_leaves_a_retryable_state():
|
||||
"""A failed ``mkdir`` must not strand the recorder in ``starting``."""
|
||||
root = _root()
|
||||
blocker = root / "not-a-directory"
|
||||
blocker.write_text("", encoding="utf-8")
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
with pytest.raises(Exception):
|
||||
recorder.start(root=blocker, session_id=1, metadata={})
|
||||
assert recorder.state == STATE_FAILED
|
||||
assert not recorder.is_recording
|
||||
assert recorder.directory is None
|
||||
assert recorder.stats().last_error
|
||||
|
||||
# Retryable: the operator fixes the path and presses record again.
|
||||
directory = recorder.start(root=root, session_id=2, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
report = recorder.stop()
|
||||
assert report.ok, report.problems
|
||||
assert Path(directory).is_dir()
|
||||
|
||||
|
||||
def test_a_stop_during_initialization_never_resumes_recording():
|
||||
"""``stop`` that lands while the files open must not be undone afterwards."""
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
original = ExperimentRecorder._write_session_json
|
||||
writing = threading.Event()
|
||||
release = threading.Event()
|
||||
calls = {"count": 0}
|
||||
|
||||
def gated(self, *, final: bool) -> None:
|
||||
calls["count"] += 1
|
||||
if calls["count"] == 1:
|
||||
# This is the writer thread's first metadata write.
|
||||
writing.set()
|
||||
release.wait(5.0)
|
||||
return original(self, final=final)
|
||||
|
||||
with patch.object(ExperimentRecorder, "_write_session_json", gated):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
assert writing.wait(5.0)
|
||||
report = recorder.stop(wait=False)
|
||||
release.set()
|
||||
thread = recorder._thread
|
||||
if thread is not None:
|
||||
thread.join(5.0)
|
||||
|
||||
# The row accepted before the stop is still written...
|
||||
with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
|
||||
rows = list(csv.DictReader(handle))
|
||||
assert len(rows) == 1, "rows accepted before the stop were dropped"
|
||||
# ...but the recording never went back to "recording", and it is finished.
|
||||
assert not recorder.is_recording
|
||||
assert recorder.state in (STATE_STOPPED, STATE_FAILED)
|
||||
assert not report.completed or report.directory is not None
|
||||
|
||||
|
||||
def test_stop_without_waiting_does_not_claim_completion():
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
original = _SessionWriter.write_sample
|
||||
|
||||
def slow(self, *args):
|
||||
started.set()
|
||||
release.wait(10.0)
|
||||
return original(self, *args)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
assert started.wait(10.0)
|
||||
report = recorder.stop(wait=False)
|
||||
try:
|
||||
assert not report.completed
|
||||
assert not report.ok
|
||||
assert recorder.report is None, (
|
||||
"an unfinished stop was cached as the terminal report"
|
||||
)
|
||||
finally:
|
||||
release.set()
|
||||
recorder._thread.join(5.0)
|
||||
assert recorder.report is not None and recorder.report.completed
|
||||
|
||||
|
||||
def test_a_failing_flush_reaches_the_state_machine_at_once():
|
||||
"""A flush failure used to live only in the writer's private counter."""
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False, flush_interval_s=0.0))
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
|
||||
class _BrokenHandle:
|
||||
def flush(self):
|
||||
raise OSError("input/output error")
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
writer = _SessionWriter(
|
||||
directory=Path(directory),
|
||||
config=recorder.config,
|
||||
session_id=1,
|
||||
calibration_id="",
|
||||
on_problem=recorder._record_failure,
|
||||
)
|
||||
writer._forces = (_BrokenHandle(), None)
|
||||
writer.maybe_flush()
|
||||
assert recorder.stats().write_errors == 1, "the flush failure was not reported"
|
||||
assert "flush failed" in recorder.stats().last_error
|
||||
|
||||
for _ in range(4):
|
||||
writer.maybe_flush()
|
||||
assert recorder.stats().failed, "repeated flush failures never failed the record"
|
||||
assert not recorder.is_recording
|
||||
recorder.stop()
|
||||
|
||||
|
||||
def test_metadata_write_failure_is_part_of_the_report():
|
||||
"""A session.json that cannot be written is part of what happened."""
|
||||
real_write_text = Path.write_text
|
||||
|
||||
def failing_write_text(self, *args, **kwargs):
|
||||
if self.name.endswith(".tmp"):
|
||||
raise OSError("read-only filesystem")
|
||||
return real_write_text(self, *args, **kwargs)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
with patch.object(Path, "write_text", failing_write_text):
|
||||
recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
report = recorder.stop()
|
||||
|
||||
assert report.stats.write_errors >= 1
|
||||
assert not report.ok, "a missing session.json was reported as a good recording"
|
||||
assert any(
|
||||
"read-only" in problem or "write error" in problem for problem in report.problems
|
||||
), report.problems
|
||||
assert report.stats.failed
|
||||
|
||||
Reference in New Issue
Block a user