Browse a long recording from disk instead of loading it into memory
What
- record/index.py: RecordingIndex streams forces.csv once and keeps only the
small things — one time, one validity flag, one frame id per row plus a byte
offset every 4096 rows (~6.5 MB per 500k rows). row_range() bisects the times,
read_rows() seeks to the nearest checkpoint and scans forward, and overview()
folds every row into its bucket during the same pass to build a per-axis
min/max envelope.
- read_rows(max_points, unit) decimates **while streaming**: each bucket keeps
the extreme row of every axis in the requested unit, so a long interval never
has to exist in memory in full and a shear spike survives a flat normal channel.
- RecordingReader: info() decimates above max_rows (default 200k) and marks
itself decimated, with total_rows and a duration taken from the frame index
rather than from the view; window()/window_all()/overview()/statistics() read
the interval being asked about. list/window semantics unchanged for small
recordings.
- force/source.py: StoreCurveSource (live memory) and RecordingCurveSource (disk
interval + cached envelope) implement one interface — extent, window,
envelope, statistics, duration_s — so ForcePlotsPanel asks a *source* instead
of knowing where the data lives.
- force_plots: reads through the source (window, envelope, statistics, extent),
which also replaces its last direct store accesses.
- playback_dialog: draws the whole-session view through window_all() in the unit
being shown (bounded by CHART_MAX_POINTS), and says so when the view is
decimated ("显示 N / 共 M 行") instead of letting the operator count points.
- experiment._run: a failure that lands while the files are opening can no longer
be overwritten by the writer's own transition to "recording" (found by a flaky
test; the state check now covers STATE_FAILED).
Why
The store releases old blocks above ~4.6 h and the replay loaded every CSV row
into Python lists before drawing anything, so neither the live browser nor the
replay could show an earlier interval of a long recording. The index makes the
disk the data source and memory a cache of what is on screen.
Verification
- shared/tests/test_history_source.py (new, 8 cases) over a 12 000-row recording:
the index reports its rows and reads a bounded interval; a large recording is
marked decimated with its true duration and total rows; a small one is not; an
interval read sees the exact rows including a single peak; a decimated window
keeps the peak; the envelope fits the whole session in ≤200 points, keeps the
spike and is cached; the disk and memory sources agree on extent, values and
statistics; the whole session can be read without holding it all. Shared
suites: 146 tests, 0 failures (three consecutive runs).
- apps/studio/tools/ui_check.py: the replay section now checks the index row
count, that an interval read spans only the interval, the envelope size and
total_rows. 324 checks, 0 failures.
- apps/studio/tests/test_ui_smoke.py: 43 tests, 36 pass; the 7 failures are the
unchanged sandbox/harness artifacts.
- D:/orisys_ws/_review_phase12/reproduce.py: all ten lines unchanged;
compileall, locale and prefs parity clean.
- apps/studio/docs/record-channel.md §8 documents the index, the reader API and
the unified source, and lists what is still hardware work.
Not in this change
- Maximum RSS / backlog / loss measurements at real resolution on the target ARM
board (needs the hardware), and range CSV export.
This commit is contained in:
ykcheungab
@@ -0,0 +1,280 @@
|
||||
"""Calibration identity and operator events as the acquisition thread sees them.
|
||||
|
||||
The review's P1-7: the application event was written before the device had loaded
|
||||
anything, the recorded ``calibration_id`` never changed mid-session, and an
|
||||
uncalibrated device could keep claiming the coefficients of an earlier session.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkAcquisitionWorker
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig
|
||||
from orisys_app_shared.record.calibration_snapshot import (
|
||||
list_calibration_snapshots,
|
||||
snapshot_name,
|
||||
store_calibration_snapshot,
|
||||
)
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
|
||||
def _root() -> Path:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
return SCRATCH
|
||||
|
||||
|
||||
class _Sensor:
|
||||
"""A device with a swappable calibration, like the SDK's Sensor."""
|
||||
|
||||
def __init__(self, calibration=None, path=None):
|
||||
self.calibration = calibration
|
||||
self.calibration_path = path
|
||||
self.applied: list[str] = []
|
||||
self.resets = 0
|
||||
|
||||
def set_calibration(self, path):
|
||||
if str(path) == "reject-me":
|
||||
raise ValueError("bad document")
|
||||
self.calibration = object()
|
||||
self.calibration_path = Path(path)
|
||||
self.applied.append(str(path))
|
||||
|
||||
def reset(self):
|
||||
self.resets += 1
|
||||
|
||||
|
||||
def _calibration_file(root: Path, name: str, body: str = '{"unit": "N"}') -> Path:
|
||||
path = root / name
|
||||
path.write_text(body, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _worker(**kwargs) -> SdkAcquisitionWorker:
|
||||
for key in ("on_calibration_applied", "on_reset_applied"):
|
||||
kwargs.setdefault(key, None)
|
||||
return SdkAcquisitionWorker(**kwargs)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ worker state
|
||||
|
||||
|
||||
def test_a_new_session_does_not_inherit_the_previous_version():
|
||||
"""An uncalibrated device must report no calibration, not the last one."""
|
||||
root = _root()
|
||||
worker = _worker()
|
||||
worker._calibration_version = "previous-session-hash"
|
||||
worker._stop.set()
|
||||
worker.start("clip.mp4", config_ref=str(_calibration_file(root, "sensor.json")))
|
||||
assert worker._calibration_version == "", (
|
||||
"the new session kept the previous session's calibration version"
|
||||
)
|
||||
worker.stop(wait=True)
|
||||
|
||||
|
||||
def test_the_applied_version_is_the_content_hash_and_carries_the_frame():
|
||||
root = _root()
|
||||
applied: list[tuple] = []
|
||||
worker = _worker(on_calibration_applied=lambda *args: applied.append(args))
|
||||
path = _calibration_file(root, "force_calibration.json")
|
||||
sensor = _Sensor()
|
||||
worker._session_id = 4
|
||||
worker.set_pending_calibration(path)
|
||||
worker._consume_pending_calibration(sensor, frame_id=17, mono_t=0.5)
|
||||
|
||||
assert len(applied) == 1
|
||||
got_path, version, frame_id, mono_t, session = applied[0]
|
||||
assert got_path == str(path)
|
||||
assert version and version == worker._calibration_version
|
||||
assert version != str(path), "the path was used as the version"
|
||||
assert (frame_id, mono_t, session) == (17, 0.5, 4)
|
||||
assert sensor.applied == [str(path)]
|
||||
|
||||
|
||||
def test_a_rejected_calibration_keeps_the_version_the_device_still_has():
|
||||
root = _root()
|
||||
applied: list[tuple] = []
|
||||
worker = _worker(on_calibration_applied=lambda *args: applied.append(args))
|
||||
sensor = _Sensor()
|
||||
worker._session_id = 1
|
||||
worker._calibration_version = "loaded-earlier"
|
||||
worker.set_pending_calibration("reject-me")
|
||||
with pytest.raises(ValueError):
|
||||
worker._consume_pending_calibration(sensor, frame_id=3, mono_t=0.1)
|
||||
assert not applied, "a rejected document was reported as applied"
|
||||
assert worker._calibration_version == "loaded-earlier", (
|
||||
"the device still has those coefficients, so the version is still true"
|
||||
)
|
||||
|
||||
|
||||
def test_a_zeroing_is_applied_by_the_worker_and_reported_with_its_frame():
|
||||
root = _root()
|
||||
events: list[tuple] = []
|
||||
worker = _worker(on_reset_applied=lambda *args: events.append(args))
|
||||
sensor = _Sensor()
|
||||
worker._session_id = 2
|
||||
worker.reset() # a request, not the zeroing itself
|
||||
assert sensor.resets == 0, "the zeroing ran on the caller's thread"
|
||||
worker._consume_pending_reset(sensor, frame_id=42, mono_t=1.4)
|
||||
assert sensor.resets == 1
|
||||
assert events == [(42, 1.4, 2)]
|
||||
# The request is consumed once.
|
||||
worker._consume_pending_reset(sensor, frame_id=43, mono_t=1.5)
|
||||
assert events == [(42, 1.4, 2)]
|
||||
|
||||
|
||||
def test_a_started_session_drops_a_stale_zeroing_request():
|
||||
root = _root()
|
||||
events: list[tuple] = []
|
||||
worker = _worker(on_reset_applied=lambda *args: events.append(args))
|
||||
worker._stop.set()
|
||||
worker.start("clip.mp4", config_ref=str(_calibration_file(root, "sensor.json")))
|
||||
sensor = _Sensor()
|
||||
worker._consume_pending_reset(sensor, frame_id=1, mono_t=0.0)
|
||||
assert sensor.resets == 0, "a reset requested before this session fired into it"
|
||||
assert not events
|
||||
worker.stop(wait=True)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- snapshots
|
||||
|
||||
|
||||
def test_a_snapshot_is_content_addressed_and_listed():
|
||||
root = _root()
|
||||
directory = root / "session"
|
||||
source = _calibration_file(root, "force_calibration_box.json", '{"a": 1}')
|
||||
name = store_calibration_snapshot(directory, source, "abcdef1234567890")
|
||||
assert name == "calibration/force_calibration_abcdef123456.json"
|
||||
assert (directory / name).is_file()
|
||||
assert (directory / name).read_text(encoding="utf-8") == '{"a": 1}'
|
||||
# Applying the same coefficients twice stores one file.
|
||||
again = store_calibration_snapshot(directory, source, "abcdef1234567890")
|
||||
assert again == name
|
||||
assert list_calibration_snapshots(directory) == [name]
|
||||
# A different calibration cannot collide with it.
|
||||
other = _calibration_file(root, "force_calibration_box.json", '{"a": 2}')
|
||||
second = store_calibration_snapshot(directory, other, "ffffff000000")
|
||||
assert second != name
|
||||
assert len(list_calibration_snapshots(directory)) == 2
|
||||
assert snapshot_name("", source).startswith("force_calibration_")
|
||||
|
||||
|
||||
def test_the_recording_metadata_names_the_coefficients_it_ran_with():
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
source = _calibration_file(root, "force_calibration_box.json", '{"a": 1}')
|
||||
name = store_calibration_snapshot(directory, source, "abcdef1234567890")
|
||||
recorder.note_calibration_snapshot(
|
||||
version="abcdef1234567890", name=name, frame_id=12, mono_t=0.4
|
||||
)
|
||||
recorder.log_event(
|
||||
"calibration_applied", name, mono_t=0.4, frame_id=12
|
||||
)
|
||||
recorder.stop()
|
||||
|
||||
meta = json.loads((Path(directory) / "session.json").read_text(encoding="utf-8"))
|
||||
snapshots = meta["calibration_snapshots"]
|
||||
assert snapshots and snapshots[0]["version"] == "abcdef1234567890"
|
||||
assert snapshots[0]["file"] == name
|
||||
assert snapshots[0]["frame_id"] == 12
|
||||
assert snapshots[0]["mono_t"] == pytest.approx(0.4)
|
||||
event = next(e for e in meta["events"] if e["kind"] == "calibration_applied")
|
||||
assert event["frame_id"] == 12
|
||||
assert event["mono_t"] == pytest.approx(0.4), "the event lost its acquisition time"
|
||||
assert event["at"], "the event lost its wall clock"
|
||||
|
||||
# The CSV carries the same acquisition time in the session's relative base.
|
||||
# An event that predates the first frame has no base yet, so its relative
|
||||
# time is empty — the absolute (session-relative) time is in the metadata.
|
||||
with (Path(directory) / "events.csv").open(encoding="utf-8", newline="") as handle:
|
||||
import csv as _csv
|
||||
|
||||
rows = list(_csv.DictReader(handle))
|
||||
applied = next(row for row in rows if row["kind"] == "calibration_applied")
|
||||
assert applied["frame_id"] == "12"
|
||||
assert applied["t_rel_s"] == "", "an event before the first frame invented a base"
|
||||
|
||||
|
||||
def test_an_event_during_the_recording_gets_the_session_relative_time():
|
||||
import csv as _csv
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
frame = np.zeros((4, 4, 3), np.uint8)
|
||||
recorder.on_packet(
|
||||
SdkPacket(
|
||||
raw_frame=frame,
|
||||
processed_frame=frame,
|
||||
timestamp=1000.0,
|
||||
mono_t=0.5,
|
||||
fps=30.0,
|
||||
process_ms=1.0,
|
||||
frame_id=7,
|
||||
)
|
||||
)
|
||||
recorder.log_event("zeroing", "tracker reset", mono_t=0.9, frame_id=9)
|
||||
recorder.stop()
|
||||
|
||||
with (Path(directory) / "events.csv").open(encoding="utf-8", newline="") as handle:
|
||||
rows = list(_csv.DictReader(handle))
|
||||
zeroing = next(row for row in rows if row["kind"] == "zeroing")
|
||||
assert float(zeroing["t_rel_s"]) == pytest.approx(0.4, abs=1e-6)
|
||||
assert zeroing["frame_id"] == "9"
|
||||
|
||||
|
||||
def test_an_event_without_a_mono_time_still_gets_a_wall_clock():
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
recorder.log_event("operator_note", "no frame attached")
|
||||
recorder.stop()
|
||||
meta = json.loads((Path(directory) / "session.json").read_text(encoding="utf-8"))
|
||||
event = next(e for e in meta["events"] if e["kind"] == "operator_note")
|
||||
assert event["at"], "the event lost its wall clock"
|
||||
assert event["frame_id"] == -1
|
||||
|
||||
|
||||
def test_the_recorded_row_uses_the_packet_version_not_the_session_default():
|
||||
from orisys_app_shared.record.sample import to_record_sample
|
||||
|
||||
class Packet:
|
||||
frame_id = 3
|
||||
mono_t = 0.2
|
||||
wall_t = 0.2
|
||||
valid = True
|
||||
calibration_version = ""
|
||||
calibrated_axes = ("fn",)
|
||||
force_calibrated = True
|
||||
raw_available = True
|
||||
|
||||
# An empty packet version is the device saying "no calibration": the session
|
||||
# default must not be pasted in behind its back.
|
||||
assert to_record_sample(Packet(), calibration_version="session-default").calibration_version == ""
|
||||
|
||||
class LegacyPacket(Packet):
|
||||
calibration_version = None # an older producer without the field
|
||||
|
||||
# ...but a packet that has no version field at all still gets the default.
|
||||
assert (
|
||||
to_record_sample(LegacyPacket(), calibration_version="session-default").calibration_version
|
||||
== "session-default"
|
||||
)
|
||||
|
||||
class StampedPacket(Packet):
|
||||
calibration_version = "packet-hash"
|
||||
|
||||
assert (
|
||||
to_record_sample(StampedPacket(), calibration_version="session-default").calibration_version
|
||||
== "packet-hash"
|
||||
)
|
||||
@@ -8,6 +8,9 @@ model.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.force.calibration_io import (
|
||||
@@ -17,10 +20,19 @@ from orisys_app_shared.force.calibration_io import (
|
||||
build_force_calibration_v2,
|
||||
merge_samples,
|
||||
predict_channel,
|
||||
write_force_calibration,
|
||||
)
|
||||
from orisys_app_shared.force.calibration_recorder import CalibrationRecorder
|
||||
from orisys_app_shared.force.session import SessionSample
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
|
||||
def _root() -> Path:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
return SCRATCH
|
||||
|
||||
|
||||
def _point(recorder, load: float, raw: float, *, channel: str = "fn"):
|
||||
recorder.start(channel=channel, reference_load_n=load, duration_s=0.05)
|
||||
@@ -151,3 +163,84 @@ def test_a_channel_with_one_point_is_not_exported():
|
||||
store.aggregate("fn", CalSample(weight_n=1.0, raw_mean=10.0, n=10))
|
||||
with pytest.raises(ValueError):
|
||||
build_force_calibration_v2(store)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- one preparation
|
||||
|
||||
|
||||
def _session_with_duplicates_and_validation():
|
||||
recorder = CalibrationRecorder()
|
||||
points = [
|
||||
_point(recorder, 1.0, 100.0),
|
||||
_point(recorder, 2.0, 200.0),
|
||||
# A deliberate repeat of load 2.0 at a different raw level.
|
||||
_point(recorder, 2.0, 240.0),
|
||||
_point(recorder, 4.0, 400.0),
|
||||
_point(recorder, 3.0, 290.0),
|
||||
]
|
||||
recorder.set_validation(points[4], True)
|
||||
return recorder, points
|
||||
|
||||
|
||||
def test_duplicate_loads_are_merged_before_the_model_is_built():
|
||||
_recorder, _points = _session_with_duplicates_and_validation()
|
||||
prepared = _recorder.prepared()
|
||||
assert prepared.samples["fn"], "no training samples"
|
||||
loads = [sample.weight_n for sample in prepared.samples["fn"]]
|
||||
assert loads == sorted(set(loads)), f"duplicate loads were not merged: {loads}"
|
||||
assert len(loads) == 3
|
||||
merged = next(sample for sample in prepared.samples["fn"] if sample.weight_n == 2.0)
|
||||
assert merged.n == 16, "the repeat's samples were not combined"
|
||||
assert merged.raw_mean == pytest.approx((200.0 + 240.0) / 2.0)
|
||||
assert prepared.points_in_fit("fn") == 3
|
||||
|
||||
|
||||
def test_the_exported_file_predicts_exactly_what_the_panel_shows():
|
||||
"""Read the written document back and compare it with the diagnostics.
|
||||
|
||||
The panel's numbers must describe the file that is actually written, so the
|
||||
test loads the file with the SDK (not with the prepared object) and predicts
|
||||
every training and validation point from it.
|
||||
"""
|
||||
import orisys.calibration as sdk_calibration
|
||||
|
||||
recorder, points = _session_with_duplicates_and_validation()
|
||||
prepared = recorder.prepared(meta={"sensor_type": "box", "sn": "SN9"})
|
||||
assert prepared.ok
|
||||
|
||||
document = prepared.to_document()
|
||||
path = _root() / "force_calibration.json"
|
||||
write_force_calibration(document, path)
|
||||
from_file = sdk_calibration.load(path, strict=True)
|
||||
|
||||
for point in points:
|
||||
raw = float(point.loaded_stats().mean)
|
||||
predicted = float(from_file.get("fn").convert(raw))
|
||||
value = prepared.predict("fn", raw)
|
||||
assert predicted == pytest.approx(value, abs=1e-9)
|
||||
if point.is_validation:
|
||||
assert prepared.validation_errors[id(point)] == pytest.approx(
|
||||
abs(predicted - point.reference_load_n), abs=1e-9
|
||||
)
|
||||
else:
|
||||
assert prepared.residuals[id(point)] == pytest.approx(
|
||||
predicted - point.reference_load_n, abs=1e-9
|
||||
)
|
||||
|
||||
# The validation point is measured, never built in.
|
||||
assert prepared.validation_points("fn") == 1
|
||||
assert prepared.max_validation_error("fn") > 0.0
|
||||
exported_raws = sorted(point[0] for point in document["channels"]["normal"]["points"])
|
||||
assert exported_raws == [100.0, 220.0, 400.0], exported_raws
|
||||
|
||||
|
||||
def test_the_recorder_fit_and_the_preparation_agree():
|
||||
recorder, _points = _session_with_duplicates_and_validation()
|
||||
prepared = recorder.prepared()
|
||||
fit = recorder.fit("fn", prepared=prepared)
|
||||
assert fit.n_points == prepared.points_in_fit("fn")
|
||||
assert fit.breakpoints == prepared.breakpoints("fn")
|
||||
assert fit.max_residual_n == pytest.approx(prepared.max_residual("fn"))
|
||||
assert fit.validation_error_n == pytest.approx(prepared.max_validation_error("fn"))
|
||||
# ...and a fit that prepares on its own gets the same numbers.
|
||||
assert recorder.fit("fn").as_dict() == fit.as_dict()
|
||||
|
||||
@@ -31,15 +31,15 @@ def _root() -> Path:
|
||||
return SCRATCH
|
||||
|
||||
|
||||
def _recorded_point(index_offset: float = 0.0):
|
||||
def _recorded_point(load: float = 2.0, offset: float = 0.0):
|
||||
recorder = CalibrationRecorder()
|
||||
recorder.start(channel="fn", reference_load_n=2.0, duration_s=0.05)
|
||||
recorder.start(channel="fn", reference_load_n=load, duration_s=0.05)
|
||||
for index in range(6):
|
||||
recorder.feed(
|
||||
SessionSample(
|
||||
index,
|
||||
index * 1.0 / 30.0,
|
||||
fn_raw=float(index) + index_offset,
|
||||
fn_raw=float(index) + offset,
|
||||
fx_raw=1.0,
|
||||
fy_raw=2.0,
|
||||
fn_n=float(index) * 10.0,
|
||||
@@ -126,9 +126,9 @@ def test_a_newer_format_is_loaded_with_a_warning():
|
||||
|
||||
def test_the_recorder_can_reopen_a_saved_session():
|
||||
points = [
|
||||
_recorded_point(),
|
||||
_recorded_point(5.0),
|
||||
_recorded_point(9.0),
|
||||
_recorded_point(1.0),
|
||||
_recorded_point(2.0, 5.0),
|
||||
_recorded_point(3.0, 9.0),
|
||||
]
|
||||
path = write_calibration_session(_root() / "session.json", points)
|
||||
restored = read_calibration_session(path).points
|
||||
@@ -141,3 +141,76 @@ def test_the_recorder_can_reopen_a_saved_session():
|
||||
fit = recorder.fit("fn")
|
||||
assert fit.validation_points == 1
|
||||
assert fit.n_points == 2
|
||||
|
||||
|
||||
def test_reopened_points_at_one_load_are_one_calibrated_load():
|
||||
"""Three points at the same reference load are one load, not a curve.
|
||||
|
||||
They are merged (n-weighted) before the build, so a session that only ever
|
||||
sampled one load cannot produce a model — which is the honest answer.
|
||||
"""
|
||||
recorder = CalibrationRecorder()
|
||||
recorder.load_points(
|
||||
[_recorded_point(2.0), _recorded_point(2.0, 5.0), _recorded_point(2.0, 9.0)],
|
||||
replace=True,
|
||||
)
|
||||
prepared = recorder.prepared()
|
||||
assert len(prepared.training) == 3
|
||||
assert len(prepared.samples["fn"]) == 1
|
||||
assert prepared.samples["fn"][0].n == 15, "the valid counts were not summed"
|
||||
assert prepared.model is None, "a single load produced a model"
|
||||
assert recorder.fit("fn").n_points == 1
|
||||
|
||||
|
||||
# ------------------------------------------------------- reopened diagnostics
|
||||
|
||||
|
||||
def test_a_reopened_record_keeps_its_device_identity():
|
||||
"""Device A's record stays A's, whatever is connected now."""
|
||||
points = [_recorded_point(1.0), _recorded_point(2.0, 5.0)]
|
||||
path = write_calibration_session(
|
||||
_root() / "session.json",
|
||||
points,
|
||||
meta={
|
||||
"sensor_type": "box",
|
||||
"sn": "SN-A",
|
||||
"config_ref": "D:/lab/deviceA/sensor.json",
|
||||
},
|
||||
)
|
||||
record = read_calibration_session(path)
|
||||
identity = record.identity()
|
||||
assert identity["sensor_type"] == "box"
|
||||
assert identity["sn"] == "SN-A"
|
||||
assert identity["config_ref"] == "D:/lab/deviceA/sensor.json"
|
||||
assert identity["created_at"] == record.created_at
|
||||
assert "SN-A" in record.describe()
|
||||
|
||||
|
||||
def test_the_three_axis_trace_is_rebuilt_from_the_samples():
|
||||
"""A record whose trace was never written must still be inspectable."""
|
||||
import json
|
||||
|
||||
point = _recorded_point(1.0)
|
||||
document = build_session_document([point])
|
||||
# Simulate a document that only kept the audit series (or an older writer
|
||||
# that stored a truncated trace).
|
||||
document["points"][0]["trace"] = {}
|
||||
document["points"][0]["duration_s"] = 0.0
|
||||
path = _root() / "session.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
|
||||
restored = read_calibration_session(path).points[0]
|
||||
assert set(restored.trace) == {"fn", "fx", "fy"}, restored.trace
|
||||
assert len(restored.trace["fn"]) == 5 # the valid samples
|
||||
assert restored.trace_dt_s == pytest.approx(1.0 / 30.0, abs=1e-6)
|
||||
assert restored.duration_s == pytest.approx(5.0 / 30.0, abs=1e-6)
|
||||
|
||||
|
||||
def test_a_stored_trace_is_not_overwritten_on_reopen():
|
||||
point = _recorded_point(1.0)
|
||||
document = build_session_document([point])
|
||||
document["points"][0]["trace"] = {"fn": [1.0, 2.0]}
|
||||
path = _root() / "session.json"
|
||||
path.write_text(json.dumps(document), encoding="utf-8")
|
||||
restored = read_calibration_session(path).points[0]
|
||||
assert restored.trace["fn"] == [1.0, 2.0]
|
||||
|
||||
@@ -16,7 +16,12 @@ from unittest.mock import patch
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.record.depth import DepthRecomputer
|
||||
from orisys_app_shared.record.depth import (
|
||||
STATE_APPROXIMATE,
|
||||
STATE_EXACT,
|
||||
STATE_PREROLL,
|
||||
DepthRecomputer,
|
||||
)
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
@@ -36,6 +41,12 @@ class _CountingSensor:
|
||||
_CountingSensor.opened.append({"args": args, "kwargs": kwargs})
|
||||
self.n = 0
|
||||
self.chain_calls = 0
|
||||
self.initial_img = "reference-frame"
|
||||
# The real Sensor opens its source in the constructor, so the first
|
||||
# source is part of its history too.
|
||||
self.vid_src = str(args[0]) if args else ""
|
||||
self.sources: list[str] = [self.vid_src] if self.vid_src else []
|
||||
self.resets = 0
|
||||
self.info = SimpleNamespace(DEPTH=1)
|
||||
|
||||
def get_img(self):
|
||||
@@ -48,9 +59,17 @@ class _CountingSensor:
|
||||
def compute_contact(self):
|
||||
self.chain_calls += 1
|
||||
|
||||
def reset(self):
|
||||
self.resets += 1
|
||||
|
||||
def read_info(self, _channel):
|
||||
return np.array([self.n])
|
||||
|
||||
def _open_video_source(self, _override, _box):
|
||||
self.sources.append(str(self.vid_src))
|
||||
# A real reopen restarts the file from its first frame.
|
||||
self.n = 0
|
||||
|
||||
def disconnect(self):
|
||||
return None
|
||||
|
||||
@@ -116,13 +135,18 @@ def test_stepping_continues_from_the_current_position():
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_stepping_backwards_reopens_the_segment():
|
||||
def test_stepping_backwards_reopens_the_source_but_keeps_the_processor():
|
||||
_CountingSensor.opened.clear()
|
||||
computer = DepthRecomputer(_reader(_session("/cfg/sensor.json")))
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
assert computer._depth_from_segment(Path("raw_0001.mp4"), 5).tolist() == [5]
|
||||
# Reading an earlier frame restarts the file; the processor object (and
|
||||
# therefore the tracker) is not thrown away.
|
||||
assert computer._depth_from_segment(Path("raw_0001.mp4"), 2).tolist() == [2]
|
||||
assert len(_CountingSensor.opened) == 2, "seeking backwards needs a fresh source"
|
||||
sensor = computer._sensor._sensor
|
||||
assert len(_CountingSensor.opened) == 1, "the processor was recreated"
|
||||
assert len(sensor.sources) == 2
|
||||
assert all(source.endswith("raw_0001.mp4") for source in sensor.sources)
|
||||
computer.close()
|
||||
|
||||
|
||||
@@ -165,11 +189,44 @@ def test_the_plan_starts_at_the_last_reset_in_the_same_segment():
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[8])
|
||||
assert plan.stop_at == 9
|
||||
assert plan.anchor == "reset"
|
||||
assert plan.state == STATE_EXACT
|
||||
assert plan.reset_frame_id == 6
|
||||
assert len(plan.steps) == 1
|
||||
step = plan.steps[0]
|
||||
assert step.source == reader.directory / "raw_0001.mp4"
|
||||
assert step.to_frame == 9
|
||||
assert step.reset_after == 7, "the reset boundary is not re-applied by the plan"
|
||||
assert plan.frames_to_read == 9
|
||||
assert plan.preroll is None
|
||||
assert plan.warnings == ()
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_reset_that_is_not_in_the_video_is_not_a_recovered_state():
|
||||
"""Finding the event is not the same as being able to start there."""
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1,
|
||||
segment_frame=index + 1,
|
||||
in_video=index != 6, # the reset frame never reached the video
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(10)
|
||||
]
|
||||
reader = _reader(_session("/cfg/sensor.json"), frames=frames)
|
||||
reader.last_reset_frame = lambda frame_id: 6
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[8])
|
||||
assert plan.anchor != "reset"
|
||||
assert plan.state == STATE_APPROXIMATE
|
||||
assert "reset_frame_not_in_video" in plan.warnings
|
||||
assert "state_approximate" in plan.warnings
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_the_plan_uses_the_pre_roll_for_the_start_of_a_recording():
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
@@ -195,6 +252,8 @@ def test_the_plan_uses_the_pre_roll_for_the_start_of_a_recording():
|
||||
plan = computer.plan_for(frames[3])
|
||||
assert plan.preroll is not None
|
||||
assert plan.preroll_frames == 30
|
||||
assert plan.anchor == "preroll"
|
||||
assert plan.state == STATE_PREROLL
|
||||
assert plan.warnings == ()
|
||||
computer.close()
|
||||
|
||||
@@ -215,10 +274,127 @@ def test_a_segment_without_history_is_flagged_approximate():
|
||||
reader = _reader(_session("/cfg/sensor.json"), frames=frames)
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[2])
|
||||
assert plan.state == STATE_APPROXIMATE
|
||||
assert plan.anchor == "segment_start"
|
||||
assert "state_approximate" in plan.warnings
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_reset_in_an_earlier_segment_is_carried_across_by_swapping_the_source():
|
||||
"""The processor survives the swap, so the state *can* cross a segment."""
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1 if index < 3 else 2,
|
||||
segment_frame=index + 1 if index < 3 else index - 2,
|
||||
in_video=True,
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(6)
|
||||
]
|
||||
reader = _reader(
|
||||
_session("/cfg/sensor.json"),
|
||||
segments=("raw_0001.mp4", "raw_0002.mp4"),
|
||||
frames=frames,
|
||||
)
|
||||
reader.last_reset_frame = lambda frame_id: 1
|
||||
computer = DepthRecomputer(reader)
|
||||
plan = computer.plan_for(frames[5])
|
||||
assert plan.anchor == "reset"
|
||||
assert plan.state == STATE_EXACT
|
||||
assert "state_across_segment" not in plan.warnings
|
||||
assert [step.source.name for step in plan.steps] == ["raw_0001.mp4", "raw_0002.mp4"]
|
||||
assert plan.steps[0].reset_after == 2
|
||||
assert plan.steps[1].to_frame == 3
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_replay_that_would_read_too_far_back_is_reported_not_attempted():
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1,
|
||||
segment_frame=index + 1,
|
||||
in_video=True,
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(20)
|
||||
]
|
||||
reader = _reader(_session("/cfg/sensor.json"), frames=frames)
|
||||
reader.last_reset_frame = lambda frame_id: 1
|
||||
computer = DepthRecomputer(reader)
|
||||
with patch("orisys_app_shared.record.depth.MAX_REPLAY_FRAMES", 5):
|
||||
plan = computer.plan_for(frames[10])
|
||||
assert "state_too_far_back" in plan.warnings
|
||||
assert plan.state == STATE_APPROXIMATE
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_the_same_frame_is_answered_from_the_cache():
|
||||
"""A second request for the frame just read is a cache hit, not None."""
|
||||
_CountingSensor.opened.clear()
|
||||
computer = DepthRecomputer(_reader(_session("/cfg/sensor.json")))
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
first = computer._depth_from_segment(Path("raw_0001.mp4"), 3)
|
||||
second = computer._depth_from_segment(Path("raw_0001.mp4"), 3)
|
||||
sensor = computer._sensor._sensor
|
||||
assert first.tolist() == [3]
|
||||
assert second is not None and second.tolist() == [3]
|
||||
assert sensor.n == 3, "the cached answer re-read the stream"
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_the_processor_survives_a_pre_roll_to_segment_switch():
|
||||
"""Only the video source is replaced; the tracker state must survive."""
|
||||
_CountingSensor.opened.clear()
|
||||
reader = _reader(_session("/cfg/sensor.json"))
|
||||
preroll = reader.directory / "preroll_0001.mp4"
|
||||
preroll.write_bytes(b"")
|
||||
computer = DepthRecomputer(reader)
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
computer._feed(preroll, 2)
|
||||
sensor = computer._sensor._sensor
|
||||
assert sensor.n == 2, "the pre-roll frames were not fed"
|
||||
# Feed the segment: same processor, new source, position restarted.
|
||||
computer._feed(reader.directory / "raw_0001.mp4", 1)
|
||||
assert len(_CountingSensor.opened) == 1, "the processor was recreated"
|
||||
assert computer._sensor._sensor is sensor
|
||||
assert len(sensor.sources) == 2
|
||||
assert sensor.sources[0].endswith("preroll_0001.mp4")
|
||||
assert sensor.sources[1].endswith("raw_0001.mp4")
|
||||
assert sensor.initial_img == "reference-frame", "the tracker reference was reset"
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_reset_boundary_is_re_applied_during_the_replay():
|
||||
"""The live zeroing cleared the tracker; the replay must do the same."""
|
||||
_CountingSensor.opened.clear()
|
||||
frames = [
|
||||
SimpleNamespace(
|
||||
frame_id=index,
|
||||
t_rel_s=index / 30.0,
|
||||
segment=1,
|
||||
segment_frame=index + 1,
|
||||
in_video=True,
|
||||
dropped=False,
|
||||
)
|
||||
for index in range(10)
|
||||
]
|
||||
reader = _reader_with_config(frames)
|
||||
reader.last_reset_frame = lambda frame_id: 6
|
||||
reader.frame_at = lambda _t: frames[8]
|
||||
computer = DepthRecomputer(reader)
|
||||
with patch("orisys.Sensor", _CountingSensor):
|
||||
result = computer.depth_at(frames[8].t_rel_s)
|
||||
sensor = computer._sensor._sensor
|
||||
assert result.ok, result.error
|
||||
assert sensor.resets == 1, "the reset boundary was not re-applied"
|
||||
assert result.warnings == (), result.warnings
|
||||
computer.close()
|
||||
|
||||
|
||||
def test_a_dropped_frame_is_reported_not_substituted():
|
||||
"""The index marks the hole; replay must say so instead of showing a neighbour."""
|
||||
frames = [
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Display unit selection and the per-axis drawing rule.
|
||||
|
||||
Two defects are covered here:
|
||||
|
||||
* the live charts chose their unit from the device-level ``force_calibrated``
|
||||
flag, so a device with only its normal-force axis calibrated either showed
|
||||
every axis in raw counts or showed raw counts inside newton panels;
|
||||
* the drawing rule asked for **all three** axes to be finite, so a valid FN
|
||||
disappeared from the chart whenever FX/FY had no calibrated value.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
from orisys_app_shared.force import select_display_forces
|
||||
from orisys_app_shared.force.session import SessionSample, SessionStore
|
||||
from orisys_app_shared.record.sample import to_record_sample
|
||||
|
||||
NAN = float("nan")
|
||||
|
||||
|
||||
def _packet(**overrides) -> SdkPacket:
|
||||
frame = np.zeros((4, 4, 3), np.uint8)
|
||||
kwargs = dict(
|
||||
raw_frame=frame,
|
||||
processed_frame=frame,
|
||||
timestamp=1000.0,
|
||||
fps=30.0,
|
||||
process_ms=1.0,
|
||||
normal_force=1.5,
|
||||
shear_x=0.25,
|
||||
shear_y=-0.25,
|
||||
normal_force_raw=123.0,
|
||||
shear_x_raw=2.0,
|
||||
shear_y_raw=-2.0,
|
||||
force_calibrated=True,
|
||||
calibrated_axes=("fn", "fx", "fy"),
|
||||
frame_id=0,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return SdkPacket(**kwargs)
|
||||
|
||||
|
||||
def test_newton_mode_marks_uncalibrated_axes_unavailable():
|
||||
sample = to_record_sample(_packet(calibrated_axes=("fn",)))
|
||||
display = select_display_forces(units_pref="newton", sample=sample)
|
||||
assert display.unit == "N"
|
||||
assert display.values[0] == pytest.approx(1.5)
|
||||
assert math.isnan(display.values[1]) and math.isnan(display.values[2])
|
||||
assert display.available == (True, False, False)
|
||||
assert display.unavailable_axes() == ("fx", "fy")
|
||||
assert not display.using_raw_fallback
|
||||
|
||||
|
||||
def test_raw_mode_shows_the_raw_counts_for_every_axis():
|
||||
sample = to_record_sample(_packet(calibrated_axes=("fn",)))
|
||||
display = select_display_forces(units_pref="raw", sample=sample)
|
||||
assert display.unit == "raw"
|
||||
assert display.values == (123.0, 2.0, -2.0)
|
||||
assert display.available == (True, True, True)
|
||||
assert not display.using_raw_fallback
|
||||
|
||||
|
||||
def test_an_uncalibrated_device_falls_back_to_raw_and_says_so():
|
||||
sample = to_record_sample(
|
||||
_packet(force_calibrated=False, calibrated_axes=())
|
||||
)
|
||||
display = select_display_forces(units_pref="newton", sample=sample)
|
||||
assert display.unit == "raw"
|
||||
assert display.using_raw_fallback, "the fallback must be announceable"
|
||||
assert display.requested_unit == "N"
|
||||
assert display.values == (123.0, 2.0, -2.0)
|
||||
|
||||
|
||||
def test_a_failed_readout_has_no_available_axis():
|
||||
sample = to_record_sample(
|
||||
_packet(valid=False, invalid_reason="compute failed", stale_forces=True)
|
||||
)
|
||||
for unit in ("N", "raw"):
|
||||
display = select_display_forces(units_pref=unit, sample=sample)
|
||||
assert display.available == (False, False, False)
|
||||
assert all(math.isnan(value) for value in display.values)
|
||||
|
||||
|
||||
def _store_with_one_calibrated_axis() -> SessionStore:
|
||||
store = SessionStore()
|
||||
for index in range(5):
|
||||
store.append(
|
||||
SessionSample(
|
||||
index,
|
||||
index / 30.0,
|
||||
fn_raw=100.0 + index,
|
||||
fx_raw=2.0,
|
||||
fy_raw=-2.0,
|
||||
fn_n=1.0 + index * 0.1,
|
||||
fx_n=NAN,
|
||||
fy_n=NAN,
|
||||
valid=True,
|
||||
calibrated_axes=("fn",),
|
||||
)
|
||||
)
|
||||
return store
|
||||
|
||||
|
||||
def test_a_valid_axis_is_drawn_even_when_the_others_have_no_value():
|
||||
"""The all-axes rule hid a good FN because FX/FY were uncalibrated."""
|
||||
slice_ = _store_with_one_calibrated_axis().data()
|
||||
times, values, mask = slice_.plottable("N")
|
||||
assert mask.shape == (5, 3)
|
||||
assert mask[:, 0].all(), "the calibrated axis was hidden"
|
||||
assert not mask[:, 1].any()
|
||||
assert not mask[:, 2].any()
|
||||
|
||||
fn_times, fn_values = slice_.plottable_axis("N", 0)
|
||||
assert fn_times.size == 5 == fn_values.size
|
||||
assert float(fn_values.max()) == pytest.approx(1.4)
|
||||
fx_times, _fx_values = slice_.plottable_axis("N", 1)
|
||||
assert fx_times.size == 0
|
||||
|
||||
|
||||
def test_the_raw_triplet_is_still_complete_for_the_same_samples():
|
||||
slice_ = _store_with_one_calibrated_axis().data()
|
||||
for index in range(3):
|
||||
times, values = slice_.plottable_axis("raw", index)
|
||||
assert times.size == 5
|
||||
assert np.isfinite(values).all()
|
||||
|
||||
|
||||
def test_an_invalid_sample_is_excluded_from_every_axis():
|
||||
store = _store_with_one_calibrated_axis()
|
||||
store.append(SessionSample(9, 9.0, fn_n=NAN, valid=False))
|
||||
slice_ = store.data()
|
||||
_times, _values, mask = slice_.plottable("N")
|
||||
assert not mask[-1].any(), "an invalid sample reached a drawing path"
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Long-history browsing: interval reads, envelopes and one curve interface.
|
||||
|
||||
The replay used to load every row of a recording before drawing anything, so a
|
||||
multi-hour session could not be shown at all. These tests pin the two things that
|
||||
make it possible — an interval read bounded by the interval, and a whole-session
|
||||
envelope built while streaming the file once — and that the charts can read from
|
||||
either a session store or a recording through the same interface.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
from orisys_app_shared.force.session import SessionSample, SessionStore
|
||||
from orisys_app_shared.force.source import RecordingCurveSource, StoreCurveSource
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig, RecordingReader
|
||||
from orisys_app_shared.record.index import RecordingIndex
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch_history"
|
||||
|
||||
ROWS = 12_000
|
||||
SPIKE_ROW = 9_000
|
||||
SPIKE_VALUE = 999.0
|
||||
|
||||
|
||||
def _root() -> Path:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
return SCRATCH
|
||||
|
||||
|
||||
def _packet(index: int) -> SdkPacket:
|
||||
frame = np.zeros((4, 4, 3), np.uint8)
|
||||
fx = SPIKE_VALUE if index == SPIKE_ROW else 0.5
|
||||
return SdkPacket(
|
||||
raw_frame=frame,
|
||||
processed_frame=frame,
|
||||
timestamp=1000.0 + index / 30.0,
|
||||
mono_t=index / 30.0,
|
||||
fps=30.0,
|
||||
process_ms=1.0,
|
||||
normal_force=1.0,
|
||||
shear_x=fx,
|
||||
shear_y=-0.5,
|
||||
normal_force_raw=10.0 + (index % 7) * 0.01,
|
||||
shear_x_raw=fx,
|
||||
shear_y_raw=-0.5,
|
||||
force_unit="N",
|
||||
force_calibrated=True,
|
||||
frame_id=index,
|
||||
)
|
||||
|
||||
|
||||
def _recording() -> Path:
|
||||
"""A recording of ``ROWS`` rows with no drops: the writer is allowed to keep up."""
|
||||
import time
|
||||
|
||||
root = _root()
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
directory = recorder.start(root=root, session_id=1, metadata={})
|
||||
for index in range(ROWS):
|
||||
recorder.on_packet(_packet(index))
|
||||
if index % 256 == 0:
|
||||
# Feeding 12000 rows in a tight loop would (correctly) overflow the
|
||||
# ordered stream and be recorded as gaps; this test is about history
|
||||
# browsing, so let the writer drain.
|
||||
deadline = time.monotonic() + 5.0
|
||||
while recorder._queue.qsize() > 512 and time.monotonic() < deadline:
|
||||
time.sleep(0.001)
|
||||
report = recorder.stop()
|
||||
assert report.stats.dropped_data == 0, report.problems
|
||||
assert report.stats.frames == ROWS
|
||||
return Path(directory)
|
||||
|
||||
|
||||
def test_the_index_reads_only_the_interval_it_is_asked_for():
|
||||
directory = _recording()
|
||||
index = RecordingIndex(directory)
|
||||
index.build()
|
||||
assert index.rows == ROWS
|
||||
assert index.duration_s() == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
|
||||
|
||||
# The whole range keeps every row, including the very first one.
|
||||
whole = index.read_rows(0, ROWS)
|
||||
assert len(whole) == ROWS
|
||||
assert whole.frame_ids[0] == 0
|
||||
assert whole.frame_ids[-1] == ROWS - 1
|
||||
|
||||
start, stop = index.row_range(300.0, 300.5)
|
||||
# One row before the interval is included on purpose: a cursor exactly on a
|
||||
# sample still wants the row it follows.
|
||||
assert 15 <= stop - start <= 20, (start, stop)
|
||||
window = index.read_rows(start, stop)
|
||||
assert 15 <= len(window) <= 20
|
||||
assert window.times[0] >= 300.0 - 1 / 30.0
|
||||
assert window.times[-1] <= 300.5
|
||||
|
||||
|
||||
def test_a_large_recording_is_loaded_as_a_marked_decimated_view():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
info = reader.info()
|
||||
assert info.decimated, "a 12000-row recording was loaded in full"
|
||||
assert info.total_rows == ROWS
|
||||
assert len(info.times) <= 500
|
||||
# The duration comes from the frame index, never from the decimated view.
|
||||
assert info.duration_s == pytest.approx((ROWS - 1) / 30.0, abs=0.05)
|
||||
assert len(info.frames) == ROWS
|
||||
|
||||
|
||||
def test_a_small_recording_is_not_decimated():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=50_000)
|
||||
info = reader.info()
|
||||
assert not info.decimated
|
||||
assert len(info.times) == ROWS
|
||||
assert info.total_rows == ROWS
|
||||
|
||||
|
||||
def test_an_interval_read_sees_every_row_including_a_single_peak():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
spike_t = SPIKE_ROW / 30.0
|
||||
window = reader.window(spike_t - 0.05, spike_t + 0.05)
|
||||
assert window.times.size >= 3
|
||||
assert float(np.nanmax(window.value_for_unit("N")[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
# ...and it is the *exact* interval, not the decimated whole-session view.
|
||||
assert not window.decimated
|
||||
assert window.times[0] >= spike_t - 0.1
|
||||
|
||||
|
||||
def test_a_decimated_window_still_keeps_the_peak():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
window = reader.window(0.0, (ROWS - 1) / 30.0, max_points=300, unit="N")
|
||||
assert window.decimated
|
||||
assert len(window) <= 600
|
||||
assert float(np.nanmax(window.value_for_unit("N")[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
|
||||
|
||||
def test_the_envelope_covers_the_whole_session_in_few_points():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
overview = reader.overview(unit="N", buckets=200)
|
||||
assert 0 < len(overview) <= 200
|
||||
assert not np.isnan(overview.highs[:, 1]).all()
|
||||
# The spike is inside a bucket the strip would otherwise skip.
|
||||
assert float(np.nanmax(overview.highs[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
assert float(overview.counts.sum()) > 0
|
||||
# Cached: asking again does not re-stream the file.
|
||||
assert reader.overview(unit="N", buckets=200) is overview
|
||||
|
||||
|
||||
def test_the_two_sources_answer_the_same_questions():
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=ROWS + 1)
|
||||
recording = RecordingCurveSource(reader)
|
||||
|
||||
store = SessionStore()
|
||||
for index in range(ROWS):
|
||||
sample = SessionSample(
|
||||
index,
|
||||
index / 30.0,
|
||||
fn_raw=10.0 + (index % 7) * 0.01,
|
||||
fx_raw=SPIKE_VALUE if index == SPIKE_ROW else 0.5,
|
||||
fy_raw=-0.5,
|
||||
fn_n=1.0,
|
||||
fx_n=SPIKE_VALUE if index == SPIKE_ROW else 0.5,
|
||||
fy_n=-0.5,
|
||||
)
|
||||
store.append(sample)
|
||||
memory = StoreCurveSource(store)
|
||||
|
||||
for source in (recording, memory):
|
||||
first, last = source.extent()
|
||||
assert first == pytest.approx(0.0, abs=1e-6)
|
||||
assert last == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
|
||||
assert source.duration_s() == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
|
||||
assert recording.kind == "disk" and memory.kind == "memory"
|
||||
|
||||
# The same interval, read from disk and from memory, agrees on the values.
|
||||
# (The disk read also includes the row just before the interval.)
|
||||
disk = recording.window(200.0, 210.0, unit="N")
|
||||
cached = memory.window(200.0, 210.0, unit="N")
|
||||
inside = (disk.times >= 200.0) & (disk.times <= 210.0)
|
||||
assert inside.sum() == len(cached) > 0
|
||||
assert np.allclose(disk.values[inside], cached.values, equal_nan=True)
|
||||
# The CSV keeps six decimals; the in-memory store keeps the raw float.
|
||||
assert np.allclose(disk.times[inside], cached.times, atol=1e-6)
|
||||
|
||||
# ...and so do the statistics of that interval (the disk read may include the
|
||||
# one row that precedes it).
|
||||
a = recording.statistics(200.0, 210.0, unit="N")
|
||||
b = memory.statistics(200.0, 210.0, unit="N")
|
||||
assert abs(a.samples - b.samples) <= 1
|
||||
assert a.peaks[1] == pytest.approx(b.peaks[1])
|
||||
assert a.means[1] == pytest.approx(b.means[1], abs=1e-6)
|
||||
|
||||
|
||||
def test_the_whole_session_can_be_read_without_holding_it_all():
|
||||
"""The point of the index: peak value with a bounded number of rows held."""
|
||||
directory = _recording()
|
||||
reader = RecordingReader(directory, max_rows=500)
|
||||
overview = reader.overview(unit="N", buckets=100)
|
||||
# What a whole-session strip needs is one band per bucket, not 12000 rows.
|
||||
assert len(overview) <= 100
|
||||
assert len(reader.info().times) <= 500
|
||||
assert reader.info().total_rows == ROWS
|
||||
# A single sample can still be read exactly, on demand.
|
||||
spike_t = SPIKE_ROW / 30.0
|
||||
exact = reader.window(spike_t, spike_t + 1 / 30.0)
|
||||
assert 1 <= len(exact) <= 3, len(exact)
|
||||
assert float(np.nanmax(exact.newtons[:, 1])) == pytest.approx(SPIKE_VALUE)
|
||||
@@ -0,0 +1,105 @@
|
||||
"""The module boundaries the responsibility split is supposed to create.
|
||||
|
||||
A pure move is only worth anything if the dependency directions actually change:
|
||||
the finger worker must not need the box worker to describe a packet, the session
|
||||
file format must not need the sampling state machine, and the models must not
|
||||
import either. These assertions are cheap and they fail loudly if someone
|
||||
re-introduces the cycle.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
SHARED = Path(__file__).resolve().parents[1] / "src"
|
||||
ACQUISITION = SHARED / "orisys_app_shared" / "acquisition"
|
||||
FORCE = SHARED / "orisys_app_shared" / "force"
|
||||
RECORD = SHARED / "orisys_app_shared" / "record"
|
||||
|
||||
|
||||
def _imports(path: Path) -> set[str]:
|
||||
"""Every module name imported by ``path`` (``from x import y`` counts as x)."""
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
found: set[str] = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
found.add(alias.name)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
if node.module:
|
||||
found.add(node.module)
|
||||
# ``from . import models`` also names the submodule.
|
||||
for alias in node.names:
|
||||
found.add(f"{node.module}.{alias.name}" if node.module else alias.name)
|
||||
return found
|
||||
|
||||
|
||||
def _imports_any(path: Path, needles) -> bool:
|
||||
found = _imports(path)
|
||||
for needle in needles:
|
||||
if any(entry == needle or entry.endswith(f".{needle}") for entry in found):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def test_the_packet_model_does_not_import_a_worker():
|
||||
assert not _imports_any(
|
||||
ACQUISITION / "models.py", {"sdk_worker", "finger_worker"}
|
||||
), "acquisition.models depends on a worker"
|
||||
|
||||
|
||||
def test_the_finger_worker_takes_the_packet_from_models():
|
||||
"""It used to import the box worker just to describe a frame."""
|
||||
source = (ACQUISITION / "finger_worker.py").read_text(encoding="utf-8")
|
||||
tree = ast.parse(source)
|
||||
from_sdk_worker: set[str] = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and (node.module or "").endswith("sdk_worker"):
|
||||
from_sdk_worker.update(alias.name for alias in node.names)
|
||||
assert "SdkPacket" not in from_sdk_worker, "the finger worker still borrows the packet"
|
||||
assert "FixedCameraMode" not in from_sdk_worker
|
||||
assert "load_fixed_mode" not in from_sdk_worker
|
||||
# ...and the helpers it genuinely shares still come from there.
|
||||
assert "process_chain" in from_sdk_worker
|
||||
|
||||
|
||||
def test_the_calibration_models_do_not_import_the_sampler_or_the_session_file():
|
||||
for path in (FORCE / "models.py",):
|
||||
assert not _imports_any(
|
||||
path, {"calibration_recorder", "calibration_session", "session"}
|
||||
), f"{path.name} depends on a higher layer"
|
||||
|
||||
|
||||
def test_the_session_format_does_not_import_the_sampling_state_machine():
|
||||
assert not _imports_any(
|
||||
FORCE / "calibration_session.py", {"calibration_recorder"}
|
||||
), "the session file format still depends on the sampler"
|
||||
|
||||
|
||||
def test_the_moved_models_are_the_same_objects_everywhere():
|
||||
from orisys_app_shared.acquisition import finger_worker, models, sdk_worker
|
||||
from orisys_app_shared.force import calibration_recorder, calibration_session, models as force_models
|
||||
|
||||
assert sdk_worker.SdkPacket is models.SdkPacket is finger_worker.SdkPacket
|
||||
assert sdk_worker.FixedCameraMode is models.FixedCameraMode
|
||||
assert calibration_recorder.CalibrationPoint is force_models.CalibrationPoint
|
||||
assert calibration_recorder.AxisStats is force_models.AxisStats
|
||||
assert calibration_session.SAMPLE_FIELDS is force_models.SAMPLE_FIELDS
|
||||
# The recorder still offers the trace helpers it used to own.
|
||||
assert callable(calibration_recorder.trace_from)
|
||||
assert callable(calibration_recorder.trace_dt)
|
||||
|
||||
|
||||
def test_the_recording_pieces_are_separate_modules():
|
||||
from orisys_app_shared.record import depth, experiment
|
||||
from orisys_app_shared.record.depth_backends import BoxReplay, FingerReplay
|
||||
from orisys_app_shared.record.video_writer import VideoSegments
|
||||
|
||||
# The depth replay no longer defines its backends inline...
|
||||
assert not hasattr(depth, "BoxReplay")
|
||||
assert not hasattr(depth, "_BoxReplay")
|
||||
# ...and the writer delegates video encoding to its own component.
|
||||
assert not hasattr(experiment, "_open_segment")
|
||||
assert callable(VideoSegments.write) and callable(VideoSegments.close)
|
||||
assert BoxReplay.__name__ == "BoxReplay" and FingerReplay.__name__ == "FingerReplay"
|
||||
@@ -137,6 +137,78 @@ def test_reader_window_and_streaming_agree(recording):
|
||||
assert streamed == 61, "streaming must see every row"
|
||||
|
||||
|
||||
def test_an_empty_recording_directory_is_readable():
|
||||
"""A folder with no CSVs must open (empty), not raise an unpack error."""
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
reader = RecordingReader(SCRATCH)
|
||||
info = reader.info()
|
||||
assert info.frames == []
|
||||
assert info.times.size == 0
|
||||
assert info.duration_s == 0.0
|
||||
assert reader.frame_at(1.0) is None
|
||||
assert reader.replay_warnings(sdk_version="0.5.1")
|
||||
finally:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
|
||||
|
||||
def test_a_new_recording_is_reported_as_time_ordered(recording):
|
||||
info = RecordingReader(recording).info()
|
||||
assert info.frames_out_of_order, "the writer no longer produces ordered rows"
|
||||
assert info.forces_out_of_order
|
||||
codes = RecordingReader(recording).replay_warnings(sdk_version="0.5.1")
|
||||
assert "out_of_order" not in codes
|
||||
|
||||
|
||||
def test_an_old_out_of_order_recording_is_re_sorted_and_reported():
|
||||
"""Rows written by the pre-fix writer are not in time order.
|
||||
|
||||
``frame_at`` binary-searches these times, so an unsorted file would answer
|
||||
with the wrong frame; the reader sorts and says so instead.
|
||||
"""
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
SCRATCH.mkdir(parents=True, exist_ok=True)
|
||||
(SCRATCH / "forces.csv").write_text(
|
||||
"session_id,frame_id,t_rel_s,FN_raw,FX_raw,FY_raw,FN_N,FX_N,FY_N,valid,calibration_id\n"
|
||||
"1,0,0.0,1.0,0.1,0.1,2.0,0.1,0.1,1,h\n"
|
||||
"1,2,0.066,3.0,0.1,0.1,6.0,0.1,0.1,1,h\n" # written before frame 1
|
||||
"1,1,0.033,2.0,0.1,0.1,4.0,0.1,0.1,1,h\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(SCRATCH / "frames.csv").write_text(
|
||||
"frame_id,t_rel_s,acquired_at,segment,segment_frame,in_video,dropped\n"
|
||||
"0,0.0,2026-01-01T00:00:00.000Z,1,1,1,0\n"
|
||||
"2,0.066,2026-01-01T00:00:00.066Z,1,3,1,0\n"
|
||||
"1,0.033,2026-01-01T00:00:00.033Z,1,2,1,0\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(SCRATCH / "session.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"format_version": "1.1",
|
||||
"device": {"sensor_type": "box", "config_ref": "/cfg/sensor.json"},
|
||||
"versions": {"sdk": "0.5.1"},
|
||||
"depth": {"stored": False},
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
try:
|
||||
reader = RecordingReader(SCRATCH)
|
||||
info = reader.info()
|
||||
assert not info.frames_out_of_order
|
||||
assert not info.forces_out_of_order
|
||||
assert [frame.frame_id for frame in info.frames] == [0, 1, 2]
|
||||
assert info.times.tolist() == pytest.approx([0.0, 0.033, 0.066])
|
||||
# The binary search is only valid because the rows were sorted.
|
||||
assert reader.frame_at(0.05).frame_id == 1
|
||||
assert reader.frame_at(0.07).frame_id == 2
|
||||
assert "out_of_order" in reader.replay_warnings(sdk_version="0.5.1")
|
||||
finally:
|
||||
shutil.rmtree(SCRATCH, ignore_errors=True)
|
||||
|
||||
|
||||
def test_depth_policy_says_recompute_and_warns_about_drift(recording):
|
||||
reader = RecordingReader(recording)
|
||||
|
||||
|
||||
@@ -10,11 +10,9 @@ from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import queue
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
@@ -23,7 +21,11 @@ import pytest
|
||||
|
||||
from orisys_app_shared.acquisition.sdk_worker import SdkPacket
|
||||
from orisys_app_shared.record import ExperimentRecorder, RecordConfig
|
||||
from orisys_app_shared.record.experiment import STATE_FAILED, STATE_STOPPED, _SessionWriter
|
||||
from orisys_app_shared.record.experiment import (
|
||||
STATE_FAILED,
|
||||
STATE_STOPPED,
|
||||
_SessionWriter,
|
||||
)
|
||||
|
||||
SCRATCH = Path(__file__).resolve().parent / "_scratch"
|
||||
|
||||
@@ -189,23 +191,30 @@ def test_a_full_queue_keeps_the_rows_and_sheds_the_video():
|
||||
|
||||
|
||||
def test_a_lost_row_is_recorded_as_an_index_gap():
|
||||
"""A hole in the data must be visible, not silently bridged in replay."""
|
||||
"""A hole in the data must be visible, not silently bridged in replay.
|
||||
|
||||
The row is only given up when the ordered stream itself is full; the gap
|
||||
marker then carries the stream position, so it lands in the index where the
|
||||
hole actually is instead of after newer frames.
|
||||
"""
|
||||
original = _SessionWriter.write_sample
|
||||
calls = {"count": 0}
|
||||
|
||||
def slow(self, *args):
|
||||
calls["count"] += 1
|
||||
if calls["count"] <= 2:
|
||||
# Stall just long enough for the queue and the tiny backlog to fill.
|
||||
# Stall just long enough for the stream to pass the (tiny) cap.
|
||||
time.sleep(0.3)
|
||||
return original(self, *args)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False, max_queue=16))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder._row_backlog = deque(recorder._row_backlog, maxlen=2)
|
||||
for index in range(40):
|
||||
recorder.on_packet(_packet(index))
|
||||
with patch(
|
||||
"orisys_app_shared.record.experiment.ROW_BACKLOG", 2
|
||||
):
|
||||
for index in range(40):
|
||||
recorder.on_packet(_packet(index))
|
||||
report = recorder.stop()
|
||||
|
||||
with (Path(directory) / "frames.csv").open(encoding="utf-8", newline="") as handle:
|
||||
@@ -218,6 +227,57 @@ def test_a_lost_row_is_recorded_as_an_index_gap():
|
||||
assert report.stats.dropped_gaps > 0, "the loss was not marked in the index"
|
||||
|
||||
|
||||
def test_the_frame_index_stays_in_time_order_when_rows_are_lost():
|
||||
"""The old writer drained its backlog before the queue, so the index jumped."""
|
||||
original = _SessionWriter.write_sample
|
||||
calls = {"count": 0}
|
||||
|
||||
def slow(self, *args):
|
||||
calls["count"] += 1
|
||||
if calls["count"] <= 3:
|
||||
time.sleep(0.2)
|
||||
return original(self, *args)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=True, max_queue=4))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
with patch("orisys_app_shared.record.experiment.ROW_BACKLOG", 3):
|
||||
for index in range(60):
|
||||
recorder.on_packet(_packet(index))
|
||||
recorder.stop()
|
||||
|
||||
with (Path(directory) / "frames.csv").open(encoding="utf-8", newline="") as handle:
|
||||
frame_rows = list(csv.DictReader(handle))
|
||||
times = [float(row["t_rel_s"]) for row in frame_rows]
|
||||
assert times == sorted(times), "frames.csv is not in time order"
|
||||
ids = [int(row["frame_id"]) for row in frame_rows]
|
||||
assert ids == sorted(ids), "forces.csv/frames.csv disagree on order"
|
||||
with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
|
||||
force_rows = list(csv.DictReader(handle))
|
||||
force_times = [float(row["t_rel_s"]) for row in force_rows]
|
||||
assert force_times == sorted(force_times), "forces.csv is not in time order"
|
||||
|
||||
|
||||
def test_the_video_byte_budget_is_released_when_a_frame_is_shed():
|
||||
"""A shed frame must not leak its reservation for the rest of the session."""
|
||||
recorder = ExperimentRecorder(
|
||||
RecordConfig(video=True, max_queue=16, max_queue_bytes=4096)
|
||||
)
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
# Frames of 8x8x3 = 192 bytes, so the budget holds ~21 of them; without the
|
||||
# release the counter would climb past the budget and stay there.
|
||||
for index in range(200):
|
||||
recorder.on_packet(_packet(index))
|
||||
report = recorder.stop()
|
||||
assert report.stats.frames == 200, "rows were lost to the video budget"
|
||||
assert abs(recorder._queued_bytes) < 4096, (
|
||||
f"the byte budget leaked: {recorder._queued_bytes} bytes still reserved"
|
||||
)
|
||||
with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
|
||||
rows = list(csv.DictReader(handle))
|
||||
assert len(rows) == 200
|
||||
|
||||
|
||||
def test_session_json_names_the_pre_roll_when_one_was_written():
|
||||
frames = [np.full((8, 8, 3), index, np.uint8) for index in range(3)]
|
||||
recorder = ExperimentRecorder(RecordConfig(video=True, fps=5.0))
|
||||
@@ -241,6 +301,80 @@ def test_a_data_only_recording_says_why_it_has_no_pre_roll():
|
||||
assert any("pre-roll" in problem for problem in report.problems)
|
||||
|
||||
|
||||
def test_a_stalled_encoder_leaves_a_readable_recording():
|
||||
"""Order, duration, positioning and the byte counter after a stall.
|
||||
|
||||
Stall the writer, push past the ordered stream's cap, let it recover, then
|
||||
push again and read the directory back the way the replay does. The frames
|
||||
acquired *after* the recovery must all be there: an overflow is a temporary
|
||||
loss, not the end of the recording.
|
||||
"""
|
||||
from orisys_app_shared.record import RecordingReader
|
||||
|
||||
original = _SessionWriter.write_sample
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
calls = {"count": 0}
|
||||
|
||||
def slow(self, *args):
|
||||
calls["count"] += 1
|
||||
if calls["count"] == 2:
|
||||
started.set()
|
||||
release.wait(10.0)
|
||||
return original(self, *args)
|
||||
|
||||
def drained(recorder, timeout: float = 10.0) -> bool:
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if recorder._queue.qsize() == 0:
|
||||
return True
|
||||
time.sleep(0.01)
|
||||
return False
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=True, fps=30.0, max_queue=4))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
with patch("orisys_app_shared.record.experiment.ROW_BACKLOG", 3):
|
||||
for index in range(40):
|
||||
recorder.on_packet(_packet(index))
|
||||
assert started.wait(10.0), "the stall never happened"
|
||||
release.set()
|
||||
assert drained(recorder), "the writer never caught up"
|
||||
before = recorder.stats().frames
|
||||
for index in range(40, 100):
|
||||
recorder.on_packet(_packet(index))
|
||||
time.sleep(0.005)
|
||||
assert drained(recorder), "the recovered writer never caught up"
|
||||
report = recorder.stop()
|
||||
|
||||
assert not report.stats.failed, report.problems
|
||||
assert recorder.stats().frames > before, "recording did not continue after the stall"
|
||||
|
||||
reader = RecordingReader(directory)
|
||||
info = reader.info()
|
||||
times = info.times.tolist()
|
||||
ids = [int(value) for value in info.frame_ids]
|
||||
assert times == sorted(times), "forces.csv is not in time order"
|
||||
assert ids == sorted(ids)
|
||||
frame_times = [frame.t_rel_s for frame in info.frames]
|
||||
assert frame_times == sorted(frame_times), "frames.csv is not in time order"
|
||||
|
||||
# Everything acquired after the recovery is present.
|
||||
assert set(range(40, 100)).issubset(set(ids)), (
|
||||
f"frames lost after the encoder recovered: {sorted(set(range(40, 100)) - set(ids))}"
|
||||
)
|
||||
assert report.stats.duration_s == pytest.approx(99 / 30.0, abs=0.05)
|
||||
assert info.duration_s == pytest.approx(99 / 30.0, abs=0.05)
|
||||
|
||||
# Positioning: the frame at 2 s is the frame acquired at 2 s.
|
||||
frame = reader.frame_at(2.0)
|
||||
assert frame is not None and frame.frame_id == 60, frame
|
||||
assert frame.t_rel_s == pytest.approx(2.0, abs=0.05)
|
||||
|
||||
# The byte budget is back to zero once every frame has been consumed.
|
||||
assert recorder._queued_bytes == 0, recorder._queued_bytes
|
||||
|
||||
|
||||
def test_stopping_twice_returns_the_same_report():
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
recorder.start(root=_root(), session_id=1, metadata={})
|
||||
@@ -249,3 +383,146 @@ def test_stopping_twice_returns_the_same_report():
|
||||
second = recorder.stop()
|
||||
assert first.directory == second.directory
|
||||
assert second.ok
|
||||
|
||||
|
||||
# ------------------------------------------------------------ failure handling
|
||||
|
||||
|
||||
def test_a_directory_that_cannot_be_created_leaves_a_retryable_state():
|
||||
"""A failed ``mkdir`` must not strand the recorder in ``starting``."""
|
||||
root = _root()
|
||||
blocker = root / "not-a-directory"
|
||||
blocker.write_text("", encoding="utf-8")
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
with pytest.raises(Exception):
|
||||
recorder.start(root=blocker, session_id=1, metadata={})
|
||||
assert recorder.state == STATE_FAILED
|
||||
assert not recorder.is_recording
|
||||
assert recorder.directory is None
|
||||
assert recorder.stats().last_error
|
||||
|
||||
# Retryable: the operator fixes the path and presses record again.
|
||||
directory = recorder.start(root=root, session_id=2, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
report = recorder.stop()
|
||||
assert report.ok, report.problems
|
||||
assert Path(directory).is_dir()
|
||||
|
||||
|
||||
def test_a_stop_during_initialization_never_resumes_recording():
|
||||
"""``stop`` that lands while the files open must not be undone afterwards."""
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
original = ExperimentRecorder._write_session_json
|
||||
writing = threading.Event()
|
||||
release = threading.Event()
|
||||
calls = {"count": 0}
|
||||
|
||||
def gated(self, *, final: bool) -> None:
|
||||
calls["count"] += 1
|
||||
if calls["count"] == 1:
|
||||
# This is the writer thread's first metadata write.
|
||||
writing.set()
|
||||
release.wait(5.0)
|
||||
return original(self, final=final)
|
||||
|
||||
with patch.object(ExperimentRecorder, "_write_session_json", gated):
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
assert writing.wait(5.0)
|
||||
report = recorder.stop(wait=False)
|
||||
release.set()
|
||||
thread = recorder._thread
|
||||
if thread is not None:
|
||||
thread.join(5.0)
|
||||
|
||||
# The row accepted before the stop is still written...
|
||||
with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
|
||||
rows = list(csv.DictReader(handle))
|
||||
assert len(rows) == 1, "rows accepted before the stop were dropped"
|
||||
# ...but the recording never went back to "recording", and it is finished.
|
||||
assert not recorder.is_recording
|
||||
assert recorder.state in (STATE_STOPPED, STATE_FAILED)
|
||||
assert not report.completed or report.directory is not None
|
||||
|
||||
|
||||
def test_stop_without_waiting_does_not_claim_completion():
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
original = _SessionWriter.write_sample
|
||||
|
||||
def slow(self, *args):
|
||||
started.set()
|
||||
release.wait(10.0)
|
||||
return original(self, *args)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
with patch.object(_SessionWriter, "write_sample", slow):
|
||||
recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
assert started.wait(10.0)
|
||||
report = recorder.stop(wait=False)
|
||||
try:
|
||||
assert not report.completed
|
||||
assert not report.ok
|
||||
assert recorder.report is None, (
|
||||
"an unfinished stop was cached as the terminal report"
|
||||
)
|
||||
finally:
|
||||
release.set()
|
||||
recorder._thread.join(5.0)
|
||||
assert recorder.report is not None and recorder.report.completed
|
||||
|
||||
|
||||
def test_a_failing_flush_reaches_the_state_machine_at_once():
|
||||
"""A flush failure used to live only in the writer's private counter."""
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False, flush_interval_s=0.0))
|
||||
directory = recorder.start(root=_root(), session_id=1, metadata={})
|
||||
|
||||
class _BrokenHandle:
|
||||
def flush(self):
|
||||
raise OSError("input/output error")
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
writer = _SessionWriter(
|
||||
directory=Path(directory),
|
||||
config=recorder.config,
|
||||
session_id=1,
|
||||
calibration_id="",
|
||||
on_problem=recorder._record_failure,
|
||||
)
|
||||
writer._forces = (_BrokenHandle(), None)
|
||||
writer.maybe_flush()
|
||||
assert recorder.stats().write_errors == 1, "the flush failure was not reported"
|
||||
assert "flush failed" in recorder.stats().last_error
|
||||
|
||||
for _ in range(4):
|
||||
writer.maybe_flush()
|
||||
assert recorder.stats().failed, "repeated flush failures never failed the record"
|
||||
assert not recorder.is_recording
|
||||
recorder.stop()
|
||||
|
||||
|
||||
def test_metadata_write_failure_is_part_of_the_report():
|
||||
"""A session.json that cannot be written is part of what happened."""
|
||||
real_write_text = Path.write_text
|
||||
|
||||
def failing_write_text(self, *args, **kwargs):
|
||||
if self.name.endswith(".tmp"):
|
||||
raise OSError("read-only filesystem")
|
||||
return real_write_text(self, *args, **kwargs)
|
||||
|
||||
recorder = ExperimentRecorder(RecordConfig(video=False))
|
||||
with patch.object(Path, "write_text", failing_write_text):
|
||||
recorder.start(root=_root(), session_id=1, metadata={})
|
||||
recorder.on_packet(_packet(0))
|
||||
report = recorder.stop()
|
||||
|
||||
assert report.stats.write_errors >= 1
|
||||
assert not report.ok, "a missing session.json was reported as a good recording"
|
||||
assert any(
|
||||
"read-only" in problem or "write error" in problem for problem in report.problems
|
||||
), report.problems
|
||||
assert report.stats.failed
|
||||
|
||||
Reference in New Issue
Block a user