Browse a long recording from disk instead of loading it into memory

What
- record/index.py: RecordingIndex streams forces.csv once and keeps only the
  small things — one time, one validity flag, one frame id per row plus a byte
  offset every 4096 rows (~6.5 MB per 500k rows). row_range() bisects the times,
  read_rows() seeks to the nearest checkpoint and scans forward, and overview()
  folds every row into its bucket during the same pass to build a per-axis
  min/max envelope.
- read_rows(max_points, unit) decimates **while streaming**: each bucket keeps
  the extreme row of every axis in the requested unit, so a long interval never
  has to exist in memory in full and a shear spike survives a flat normal channel.
- RecordingReader: info() decimates above max_rows (default 200k) and marks
  itself decimated, with total_rows and a duration taken from the frame index
  rather than from the view; window()/window_all()/overview()/statistics() read
  the interval being asked about. list/window semantics unchanged for small
  recordings.
- force/source.py: StoreCurveSource (live memory) and RecordingCurveSource (disk
  interval + cached envelope) implement one interface — extent, window,
  envelope, statistics, duration_s — so ForcePlotsPanel asks a *source* instead
  of knowing where the data lives.
- force_plots: reads through the source (window, envelope, statistics, extent),
  which also replaces its last direct store accesses.
- playback_dialog: draws the whole-session view through window_all() in the unit
  being shown (bounded by CHART_MAX_POINTS), and says so when the view is
  decimated ("显示 N / 共 M 行") instead of letting the operator count points.
- experiment._run: a failure that lands while the files are opening can no longer
  be overwritten by the writer's own transition to "recording" (found by a flaky
  test; the state check now covers STATE_FAILED).

Why
The store releases old blocks above ~4.6 h and the replay loaded every CSV row
into Python lists before drawing anything, so neither the live browser nor the
replay could show an earlier interval of a long recording. The index makes the
disk the data source and memory a cache of what is on screen.

Verification
- shared/tests/test_history_source.py (new, 8 cases) over a 12 000-row recording:
  the index reports its rows and reads a bounded interval; a large recording is
  marked decimated with its true duration and total rows; a small one is not; an
  interval read sees the exact rows including a single peak; a decimated window
  keeps the peak; the envelope fits the whole session in ≤200 points, keeps the
  spike and is cached; the disk and memory sources agree on extent, values and
  statistics; the whole session can be read without holding it all. Shared
  suites: 146 tests, 0 failures (three consecutive runs).
- apps/studio/tools/ui_check.py: the replay section now checks the index row
  count, that an interval read spans only the interval, the envelope size and
  total_rows. 324 checks, 0 failures.
- apps/studio/tests/test_ui_smoke.py: 43 tests, 36 pass; the 7 failures are the
  unchanged sandbox/harness artifacts.
- D:/orisys_ws/_review_phase12/reproduce.py: all ten lines unchanged;
  compileall, locale and prefs parity clean.
- apps/studio/docs/record-channel.md §8 documents the index, the reader API and
  the unified source, and lists what is still hardware work.

Not in this change
- Maximum RSS / backlog / loss measurements at real resolution on the target ARM
  board (needs the hardware), and range CSV export.

shared/src/orisys_app_shared/acquisition/finger_worker.py

@@ -14,20 +14,25 @@ from orisys_app_shared.acquisition.constants import (
     LOWPASS_CUTOFF_HZ,
 )
 from orisys_app_shared.acquisition.sinks import RecordSinkMixin
+# The packet and the geometry come from ``models``, so the finger worker does not
+# depend on the box worker; only genuinely shared helpers are imported from it.
+from orisys_app_shared.acquisition.models import (
+    FixedCameraMode,
+    SdkPacket,
+    ensure_bgr as _ensure_bgr,
+    is_file_source as _is_file_source,
+    load_fixed_mode,
+    orisys_importable,
+)
 from orisys_app_shared.acquisition.sdk_worker import (
     CAMERA_DISCONNECT_TIMEOUT_S,
     HANDOVER_TIMEOUT_S,
     STOP_JOIN_TIMEOUT_S,
     VIDEO_EOF_TIMEOUT_S,
-    FixedCameraMode,
-    SdkPacket,
-    _ensure_bgr,
-    _is_file_source,
     calibration_digest,
-    load_fixed_mode,
     loaded_calibration_version,
     loaded_path,
-    orisys_importable,
+    process_chain,
     read_forces_from_sensor,
 )
 
@@ -65,13 +70,17 @@ class FingerAcquisitionWorker(RecordSinkMixin):
         *,
         on_error: Optional[Callable[[str], None]] = None,
         on_status: Optional[Callable[[str], None]] = None,
-        on_calibration_applied: Optional[Callable[[str, str, int], None]] = None,
+        on_calibration_applied: Optional[Callable[[str, str, int, float, int], None]] = None,
+        on_reset_applied: Optional[Callable[[int, float, int], None]] = None,
     ):
         self._init_sinks()
         self._on_error = on_error
         self._on_status = on_status
-        #: Called from the acquisition thread once new coefficients are loaded.
+        #: Called from the acquisition thread once new coefficients are loaded,
+        #: with ``(path, version, frame_id, mono_t, session_id)``.
         self._on_calibration_applied = on_calibration_applied
+        #: Called from the acquisition thread once a queued zeroing took effect.
+        self._on_reset_applied = on_reset_applied
         self._thread: Optional[threading.Thread] = None
         self._stop = threading.Event()
         self._lock = threading.Lock()
@@ -86,6 +95,8 @@ class FingerAcquisitionWorker(RecordSinkMixin):
         self._end_reason: Optional[str] = None
         self._phase: str = "idle"
         self._pending_calibration: Optional[str] = None
+        #: A zeroing requested from another thread, applied by this one.
+        self._pending_reset: bool = False
         self._mono_t0: float = 0.0
         self._calibration_version: str = ""
 
@@ -129,6 +140,11 @@ class FingerAcquisitionWorker(RecordSinkMixin):
             self._latest = None
             self._end_reason = None
             self._phase = "connecting"
+            # This session starts with no calibration of its own: an uncalibrated
+            # runtime must not inherit the previous session's version, and a stale
+            # zeroing request must not fire into the new session's frames.
+            self._calibration_version = ""
+            self._pending_reset = False
         self._session_id += 1
         session = self._session_id
         self._stop.clear()
@@ -171,22 +187,54 @@ class FingerAcquisitionWorker(RecordSinkMixin):
             self._phase = "idle"
 
     def reset(self) -> None:
+        """Request a tracker reset, applied by the acquisition thread.
+
+        The runtime's tactile reset from the GUI thread would race the frame the
+        loop is processing, so the zeroing happens at a frame boundary on the
+        acquisition thread, which reports the frame and time it took effect.
+        """
         with self._lock:
-            runtime = self._runtimes.get(self._session_id)
-        if runtime is None:
-            raise RuntimeError("Finger runtime is not running")
-        runtime.tactile.reset()
+            if self._session_id <= 0:
+                raise RuntimeError("Finger runtime is not running")
+            self._pending_reset = True
+
+    def _consume_pending_reset(self, runtime, frame_id: int, mono_t: float) -> None:
+        """Apply a queued reset and report the frame it actually took effect at."""
+        with self._lock:
+            pending = self._pending_reset
+            self._pending_reset = False
+            session = self._session_id
+        if not pending or runtime is None:
+            return
+        tactile = getattr(runtime, "tactile", None)
+        target = tactile if callable(getattr(tactile, "reset", None)) else None
+        if target is None:
+            sensor = getattr(tactile, "raw_sensor", None)
+            target = sensor if callable(getattr(sensor, "reset", None)) else None
+        if target is None:
+            raise RuntimeError("runtime does not support reset")
+        target.reset()
+        if self._on_reset_applied is not None:
+            self._on_reset_applied(int(frame_id), float(mono_t), int(session))
 
     def set_pending_calibration(self, path: str | Path | None) -> None:
         """Queue a force_calibration.json path for the acquisition thread."""
         with self._lock:
             self._pending_calibration = str(path) if path is not None else None
 
-    def _consume_pending_calibration(self, sensor, frame_id: int = 0) -> None:
-        """Apply queued coefficients and report confirmation, not intent."""
+    def _consume_pending_calibration(
+        self, sensor, frame_id: int = 0, mono_t: float = 0.0
+    ) -> None:
+        """Apply queued coefficients and report confirmation, not intent.
+
+        A rejected document leaves the version alone on purpose: the sensor still
+        has the coefficients it had. What never happens is inventing a version for
+        a device that has no calibration — that is the empty string.
+        """
         with self._lock:
             path = self._pending_calibration
             self._pending_calibration = None
+            session = self._session_id
         if path is None or sensor is None:
             return
         if not hasattr(sensor, "set_calibration"):
@@ -195,7 +243,9 @@ class FingerAcquisitionWorker(RecordSinkMixin):
         version = calibration_digest(path)
         self._calibration_version = version
         if self._on_calibration_applied is not None:
-            self._on_calibration_applied(str(path), version, int(frame_id))
+            self._on_calibration_applied(
+                str(path), version, int(frame_id), float(mono_t), int(session)
+            )
 
     def _set_end_reason(self, reason: str) -> None:
         with self._lock:
@@ -287,12 +337,13 @@ class FingerAcquisitionWorker(RecordSinkMixin):
             # name those coefficients from the first frame, not the request path.
             tactile = getattr(runtime, "tactile", None)
             loaded = loaded_calibration_version(tactile) if tactile else ""
-            if loaded:
-                self._calibration_version = loaded
-                if self._on_calibration_applied is not None:
-                    self._on_calibration_applied(
-                        str(loaded_path(tactile) or ""), loaded, 0
-                    )
+            # Assigned even when empty: an uncalibrated device reports *no*
+            # calibration, never the previous session's version.
+            self._calibration_version = loaded
+            if loaded and self._on_calibration_applied is not None:
+                self._on_calibration_applied(
+                    str(loaded_path(tactile) or ""), loaded, 0, 0.0, int(session)
+                )
         except Exception as exc:
             self._set_end_reason("error")
             if self._on_error is not None and session == self._session_id:
@@ -341,20 +392,26 @@ class FingerAcquisitionWorker(RecordSinkMixin):
                 none_streak_start = None
                 tactile = runtime.tactile
                 compute_error = ""
+                # The frame's acquisition time, so an event that happens on this
+                # frame is dated when it happened rather than when it was queued.
+                frame_mono = max(0.0, time.perf_counter() - self._mono_t0)
                 try:
-                    self._consume_pending_calibration(tactile, frame_id)
+                    self._consume_pending_calibration(tactile, frame_id, frame_mono)
                 except Exception as exc:  # noqa: BLE001
                     message = f"calibration rejected: {type(exc).__name__}: {exc}"
                     if self._on_error is not None and session == self._session_id:
                         self._on_error(message)
                     compute_error = message
                 try:
-                    tactile.compute_deformation(decompose=True, check_motion=False)
-                    tactile.compute_contact()
-                except Exception as exc:  # noqa: BLE001 - reported on the packet
-                    compute_error = compute_error or (
-                        f"compute: {type(exc).__name__}: {exc}"
-                    )
+                    # Applied between two frames on this thread, then reported.
+                    self._consume_pending_reset(runtime, frame_id, frame_mono)
+                except Exception as exc:  # noqa: BLE001
+                    message = f"reset failed: {type(exc).__name__}: {exc}"
+                    if self._on_error is not None and session == self._session_id:
+                        self._on_error(message)
+                # The same chain the offline depth replay runs on the tactile.
+                chain_depth, chain_error = process_chain(tactile)
+                compute_error = compute_error or chain_error
                 t1 = time.perf_counter()
                 frame_times.append(t1)
                 cutoff = t1 - 1.0
@@ -395,7 +452,11 @@ class FingerAcquisitionWorker(RecordSinkMixin):
                     stale_forces=bool(compute_error),
                     raw_available=bool(forces.get("raw_available", True)),
                     flow=forces["flow"],
-                    depth=forces.get("depth"),
+                    depth=(
+                        forces.get("depth")
+                        if forces.get("depth") is not None
+                        else chain_depth
+                    ),
                     valid_mask=runtime.valid_mask,
                 )
                 with self._lock:

shared/src/orisys_app_shared/acquisition/models.py

@@ -0,0 +1,131 @@
+"""The data shapes shared by the acquisition workers, the record channel and the UI.
+
+They live here rather than in one of the workers so that neither worker has to
+import the other, and so the record channel can describe a packet without
+importing a device loop. The workers re-export these names for compatibility.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Optional
+
+import cv2
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass(frozen=True)
+class FixedCameraMode:
+    """The camera geometry a session runs with."""
+
+    width: int
+    height: int
+    fps: float
+    config_ref: str
+
+    def label(self) -> str:
+        return f"{self.width}x{self.height} @{self.fps:g}"
+
+
+@dataclass
+class SdkPacket:
+    """One acquired frame: the images, the force readout and its provenance."""
+
+    raw_frame: np.ndarray
+    processed_frame: np.ndarray
+    timestamp: float
+    fps: float
+    process_ms: float
+    normal_force: float = 0.0
+    shear_x: float = 0.0
+    shear_y: float = 0.0
+    normal_force_raw: float = 0.0
+    shear_x_raw: float = 0.0
+    shear_y_raw: float = 0.0
+    force_unit: str = "raw"
+    force_calibrated: bool = False
+    #: Which axes actually have a calibration curve loaded. The sensor-level
+    #: ``force_calibrated`` flag cannot express a partially calibrated device.
+    calibrated_axes: tuple[str, ...] = ()
+    session_id: int = 0
+    #: Monotonic per-session frame index; the join key for video, CSV and
+    #: calibration records.
+    frame_id: int = 0
+    #: Monotonic, session-relative acquisition time. Used for ordering so a wall
+    #: clock that steps backwards cannot reorder history. ``None`` means the
+    #: producer did not measure one, and the wall clock is used instead — a
+    #: default of ``0.0`` would have made every frame look simultaneous.
+    mono_t: Optional[float] = None
+    #: Wall clock at hand-off, for metadata only. ``None`` means unmeasured.
+    wall_t: Optional[float] = None
+    #: False when the force readout failed. Invalid packets carry NaN forces and
+    #: must never be averaged into calibration data or written as zeros.
+    valid: bool = True
+    invalid_reason: str = ""
+    #: True when the compute chain failed for this frame. The SDK's force fields
+    #: then still hold the value computed for an earlier frame, so they must not
+    #: be recorded as this frame's measurement.
+    stale_forces: bool = False
+    #: Content hash of the calibration actually in effect when this frame was
+    #: acquired; changes mid-session when new coefficients are applied.
+    calibration_version: str = ""
+    #: False when the sensor exposes no RAW channels; the raw fields are then
+    #: NaN rather than a copy of the calibrated values.
+    raw_available: bool = True
+    flow: Optional[np.ndarray] = None
+    depth: Optional[np.ndarray] = None
+    valid_mask: Optional[np.ndarray] = None
+
+
+def load_fixed_mode(config_ref: str | Path) -> FixedCameraMode:
+    """Resolve a config reference (a file path or a product name) to its geometry."""
+    ref = str(config_ref).strip()
+    path = Path(ref)
+    if path.is_file():
+        data = json.loads(path.read_text(encoding="utf-8"))
+        resolved = str(path.resolve())
+    else:
+        import orisys
+
+        data = orisys.configs.load_config(ref)
+        resolved = ref
+    camera = data.get("camera") or {}
+    raw = camera.get("resolution_raw") or [1920, 1080]
+    fps = float(camera.get("fps") or 30.0)
+    return FixedCameraMode(
+        width=int(raw[0]),
+        height=int(raw[1]),
+        fps=fps,
+        config_ref=resolved,
+    )
+
+
+def ensure_bgr(img: np.ndarray) -> np.ndarray:
+    """Three-channel BGR view of whatever the sensor hands over."""
+    if img.ndim == 2:
+        return cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
+    if img.ndim == 3 and img.shape[2] == 1:
+        return cv2.cvtColor(img[:, :, 0], cv2.COLOR_GRAY2BGR)
+    if img.ndim == 3 and img.shape[2] == 4:
+        return cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)
+    return img
+
+
+def is_file_source(video_source: str | int) -> bool:
+    """True when the source is a file (a replay) rather than a live camera."""
+    return isinstance(video_source, str)
+
+
+def orisys_importable() -> tuple[bool, str]:
+    """Whether the SDK can be imported, with the reason when it cannot."""
+    try:
+        import orisys  # noqa: F401
+
+        return True, getattr(__import__("orisys"), "__version__", "unknown")
+    except Exception as exc:  # noqa: BLE001
+        return False, str(exc)

shared/src/orisys_app_shared/acquisition/sdk_worker.py

@@ -19,6 +19,14 @@ from orisys_app_shared.acquisition.constants import (
     FORCE_SAMPLE_RATE_HZ,
     LOWPASS_CUTOFF_HZ,
 )
+from orisys_app_shared.acquisition.models import (
+    FixedCameraMode,
+    SdkPacket,
+    ensure_bgr as _ensure_bgr,
+    is_file_source as _is_file_source,
+    load_fixed_mode,
+    orisys_importable,
+)
 from orisys_app_shared.acquisition.sinks import RecordSinkMixin
 
 VIDEO_EOF_TIMEOUT_S = 0.5
@@ -29,105 +37,22 @@ STOP_JOIN_TIMEOUT_S = 5.0
 #: previous session that is still winding down before opening the device.
 HANDOVER_TIMEOUT_S = 60.0
 
+# Re-exported so ``from ...sdk_worker import SdkPacket`` keeps working; the
+# definitions live in ``models`` so neither worker depends on the other.
+__all__ = [
+    "FixedCameraMode",
+    "SdkPacket",
+    "SdkAcquisitionWorker",
+    "calibration_digest",
+    "load_fixed_mode",
+    "loaded_calibration_version",
+    "loaded_path",
+    "orisys_importable",
+    "process_chain",
+    "read_depth",
+    "read_forces_from_sensor",
+]
 
-@dataclass(frozen=True)
-class FixedCameraMode:
-    width: int
-    height: int
-    fps: float
-    config_ref: str
-
-    def label(self) -> str:
-        return f"{self.width}x{self.height} @{self.fps:g}"
-
-
-@dataclass
-class SdkPacket:
-    raw_frame: np.ndarray
-    processed_frame: np.ndarray
-    timestamp: float
-    fps: float
-    process_ms: float
-    normal_force: float = 0.0
-    shear_x: float = 0.0
-    shear_y: float = 0.0
-    normal_force_raw: float = 0.0
-    shear_x_raw: float = 0.0
-    shear_y_raw: float = 0.0
-    force_unit: str = "raw"
-    force_calibrated: bool = False
-    #: Which axes actually have a calibration curve loaded. The sensor-level
-    #: ``force_calibrated`` flag cannot express a partially calibrated device.
-    calibrated_axes: tuple[str, ...] = ()
-    session_id: int = 0
-    #: Monotonic per-session frame index; the join key for video, CSV and
-    #: calibration records.
-    frame_id: int = 0
-    #: Monotonic, session-relative acquisition time. Used for ordering so a wall
-    #: clock that steps backwards cannot reorder history. ``None`` means the
-    #: producer did not measure one, and the wall clock is used instead — a
-    #: default of ``0.0`` would have made every frame look simultaneous.
-    mono_t: Optional[float] = None
-    #: Wall clock at hand-off, for metadata only. ``None`` means unmeasured.
-    wall_t: Optional[float] = None
-    #: False when the force readout failed. Invalid packets carry NaN forces and
-    #: must never be averaged into calibration data or written as zeros.
-    valid: bool = True
-    invalid_reason: str = ""
-    #: True when the compute chain failed for this frame. The SDK's force fields
-    #: then still hold the value computed for an earlier frame, so they must not
-    #: be recorded as this frame's measurement.
-    stale_forces: bool = False
-    #: Content hash of the calibration actually in effect when this frame was
-    #: acquired; changes mid-session when new coefficients are applied.
-    calibration_version: str = ""
-    #: False when the sensor exposes no RAW channels; the raw fields are then
-    #: NaN rather than a copy of the calibrated values.
-    raw_available: bool = True
-    flow: Optional[np.ndarray] = None
-    depth: Optional[np.ndarray] = None
-    valid_mask: Optional[np.ndarray] = None
-
-
-def orisys_importable() -> tuple[bool, str]:
-    try:
-        import orisys  # noqa: F401
-
-        return True, getattr(__import__("orisys"), "__version__", "unknown")
-    except Exception as exc:  # noqa: BLE001
-        return False, str(exc)
-
-
-def load_fixed_mode(config_ref: str | Path) -> FixedCameraMode:
-    ref = str(config_ref).strip()
-    path = Path(ref)
-    if path.is_file():
-        data = json.loads(path.read_text(encoding="utf-8"))
-        resolved = str(path.resolve())
-    else:
-        import orisys
-
-        data = orisys.configs.load_config(ref)
-        resolved = ref
-    camera = data.get("camera") or {}
-    raw = camera.get("resolution_raw") or [1920, 1080]
-    fps = float(camera.get("fps") or 30.0)
-    return FixedCameraMode(
-        width=int(raw[0]),
-        height=int(raw[1]),
-        fps=fps,
-        config_ref=resolved,
-    )
-
-
-def _ensure_bgr(img: np.ndarray) -> np.ndarray:
-    if img.ndim == 2:
-        return cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
-    if img.ndim == 3 and img.shape[2] == 1:
-        return cv2.cvtColor(img[:, :, 0], cv2.COLOR_GRAY2BGR)
-    if img.ndim == 3 and img.shape[2] == 4:
-        return cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)
-    return img
 
 
 def _sensor_force_calibrated(sensor, *, config_ref: str | Path | None = None) -> bool:
@@ -147,6 +72,43 @@ def _sensor_force_calibrated(sensor, *, config_ref: str | Path | None = None) ->
         return False
 
 
+def process_chain(sensor) -> tuple[Optional[np.ndarray], str]:
+    """Run the live processing chain for one already-read frame.
+
+    ``get_img`` alone is not the pipeline: the SDK's depth map is produced by
+    ``compute_deformation`` + ``compute_contact``. The offline depth replay calls
+    this same function, so it cannot drift from what was recorded live.
+    Returns ``(depth, error)``; the depth is ``None`` when the chain failed or the
+    sensor exposes none.
+    """
+    compute_error = ""
+    compute_deformation = getattr(sensor, "compute_deformation", None)
+    compute_contact = getattr(sensor, "compute_contact", None)
+    if callable(compute_deformation) and callable(compute_contact):
+        # Same calls and arguments as the acquisition loop.
+        try:
+            compute_deformation(decompose=True, check_motion=False)
+            compute_contact()
+        except Exception as exc:  # noqa: BLE001 - reported to the caller
+            compute_error = f"compute: {type(exc).__name__}: {exc}"
+    else:
+        compute_error = f"no processing chain on {type(sensor).__name__}"
+    return read_depth(sensor), compute_error
+
+
+def read_depth(sensor) -> Optional[np.ndarray]:
+    """The current depth channel of ``sensor``, or ``None`` when unavailable."""
+    try:
+        info = sensor.info
+        if not hasattr(info, "DEPTH"):
+            return None
+        value = sensor.read_info(info.DEPTH)
+    except Exception:  # noqa: BLE001 - reported as "no depth for this frame"
+        logger.exception("Could not read the depth channel")
+        return None
+    return None if value is None else np.asarray(value)
+
+
 def read_forces_from_sensor(sensor, *, config_ref: str | Path | None = None):
     """Read one force sample.
 
@@ -284,10 +246,6 @@ def _calibrated_axes(sensor) -> tuple[str, ...]:
     return tuple(axes)
 
 
-def _is_file_source(video_source: str | int) -> bool:
-    return isinstance(video_source, str)
-
-
 def calibration_digest(path: str | Path) -> str:
     """Content hash of a calibration document — the calibration's identity.
 
@@ -352,14 +310,20 @@ class SdkAcquisitionWorker(RecordSinkMixin):
         *,
         on_error: Optional[Callable[[str], None]] = None,
         on_status: Optional[Callable[[str], None]] = None,
-        on_calibration_applied: Optional[Callable[[str, str, int], None]] = None,
+        on_calibration_applied: Optional[Callable[[str, str, int, float, int], None]] = None,
+        on_reset_applied: Optional[Callable[[int, float, int], None]] = None,
     ):
         self._init_sinks()
         self._on_error = on_error
         self._on_status = on_status
         #: Called from the acquisition thread once new coefficients are actually
-        #: loaded, with ``(path, version, frame_id)``. Confirmation, not intent.
+        #: loaded, with ``(path, version, frame_id, mono_t, session_id)``.
+        #: Confirmation, not intent — and it carries the frame and time so the
+        #: recorded event is dated when it happened.
         self._on_calibration_applied = on_calibration_applied
+        #: Called from the acquisition thread once a queued zeroing took effect,
+        #: with ``(frame_id, mono_t, session_id)``.
+        self._on_reset_applied = on_reset_applied
         self._thread: Optional[threading.Thread] = None
         self._stop = threading.Event()
         self._lock = threading.Lock()
@@ -373,6 +337,8 @@ class SdkAcquisitionWorker(RecordSinkMixin):
         self._end_reason: Optional[str] = None
         self._phase: str = "idle"
         self._pending_calibration: Optional[str] = None
+        #: A zeroing requested from another thread, applied by this one.
+        self._pending_reset: bool = False
         #: Monotonic origin for ``SdkPacket.mono_t``, reset per session.
         self._mono_t0: float = 0.0
         #: Content hash of the calibration currently in effect.
@@ -418,6 +384,11 @@ class SdkAcquisitionWorker(RecordSinkMixin):
             self._latest = None
             self._end_reason = None
             self._phase = "connecting"
+            # The previous session's coefficients say nothing about this one: an
+            # uncalibrated device must not inherit a version it never loaded, and
+            # an old reset request must not fire into the new session's frames.
+            self._calibration_version = ""
+            self._pending_reset = False
         self._session_id += 1
         session = self._session_id
         self._stop.clear()
@@ -463,28 +434,57 @@ class SdkAcquisitionWorker(RecordSinkMixin):
             self._phase = "idle"
 
     def reset(self) -> None:
+        """Request a tracker reset, applied by the acquisition thread.
+
+        ``Sensor.reset()`` from the GUI thread would race the frame the loop is
+        processing, so the actual zeroing happens at a frame boundary on the
+        acquisition thread — which then reports the frame and the time it took
+        effect. The confirmation callback is ``on_reset_applied``.
+        """
         with self._lock:
-            sensor = self._sensors.get(self._session_id)
-        if sensor is None:
-            raise RuntimeError("Sensor is not running")
-        sensor.reset()
+            if self._session_id <= 0:
+                raise RuntimeError("Sensor is not running")
+            self._pending_reset = True
+
+    def _consume_pending_reset(self, sensor, frame_id: int, mono_t: float) -> None:
+        """Apply a queued reset and report the frame it actually took effect at."""
+        with self._lock:
+            pending = self._pending_reset
+            self._pending_reset = False
+            session = self._session_id
+        if not pending or sensor is None:
+            return
+        reset = getattr(sensor, "reset", None)
+        if not callable(reset):
+            raise RuntimeError("sensor does not support reset")
+        reset()
+        if self._on_reset_applied is not None:
+            self._on_reset_applied(int(frame_id), float(mono_t), int(session))
 
     def set_pending_calibration(self, path: str | Path | None) -> None:
         """Queue a force_calibration.json path for the acquisition thread."""
         with self._lock:
             self._pending_calibration = str(path) if path is not None else None
 
-    def _consume_pending_calibration(self, sensor, frame_id: int = 0) -> None:
+    def _consume_pending_calibration(
+        self, sensor, frame_id: int = 0, mono_t: float = 0.0
+    ) -> None:
         """Apply queued coefficients and report **confirmation**, not intent.
 
         ``Sensor.set_calibration`` loads with ``strict=True`` and raises on an
         invalid document, so success is observable. The old code swallowed that
         exception, which is why the app logged "calibration applied" before
         anything had actually been loaded.
+
+        A rejection leaves the version alone on purpose: the sensor still has the
+        coefficients it had, so the previous version is still the true one. What
+        never happens is inventing a version for a device that has no calibration
+        — that is the empty string, set when the session opened.
         """
         with self._lock:
             path = self._pending_calibration
             self._pending_calibration = None
+            session = self._session_id
         if path is None or sensor is None:
             return
         if not hasattr(sensor, "set_calibration"):
@@ -493,7 +493,9 @@ class SdkAcquisitionWorker(RecordSinkMixin):
         version = calibration_digest(path)
         self._calibration_version = version
         if self._on_calibration_applied is not None:
-            self._on_calibration_applied(str(path), version, int(frame_id))
+            self._on_calibration_applied(
+                str(path), version, int(frame_id), float(mono_t), int(session)
+            )
 
     def _set_end_reason(self, reason: str) -> None:
         with self._lock:
@@ -574,12 +576,13 @@ class SdkAcquisitionWorker(RecordSinkMixin):
             # here. Without this the record would name the path passed on the
             # command line instead of the calibration that was applied.
             loaded = loaded_calibration_version(sensor)
-            if loaded:
-                self._calibration_version = loaded
-                if self._on_calibration_applied is not None:
-                    self._on_calibration_applied(
-                        str(loaded_path(sensor) or ""), loaded, 0
-                    )
+            # Assigned even when empty: an uncalibrated device must report *no*
+            # calibration, never the previous session's version.
+            self._calibration_version = loaded
+            if loaded and self._on_calibration_applied is not None:
+                self._on_calibration_applied(
+                    str(loaded_path(sensor) or ""), loaded, 0, 0.0, int(session)
+                )
             self._status(f"Streaming {mode.label()}")
         except Exception as exc:
             self._set_end_reason("error")
@@ -627,23 +630,32 @@ class SdkAcquisitionWorker(RecordSinkMixin):
 
                 none_streak_start = None
                 compute_error = ""
+                # The frame's acquisition time, so an event that happens on this
+                # frame is dated when it happened rather than when it was queued.
+                frame_mono = max(0.0, time.perf_counter() - self._mono_t0)
                 try:
                     # Applying new coefficients is reported separately from the
                     # compute chain: a rejected document is an operator error,
                     # not a numerical failure.
-                    self._consume_pending_calibration(sensor, frame_id)
+                    self._consume_pending_calibration(sensor, frame_id, frame_mono)
                 except Exception as exc:  # noqa: BLE001
                     message = f"calibration rejected: {type(exc).__name__}: {exc}"
                     if self._on_error is not None and session == self._session_id:
                         self._on_error(message)
                     compute_error = message
                 try:
-                    sensor.compute_deformation(decompose=True, check_motion=False)
-                    sensor.compute_contact()
-                except Exception as exc:  # noqa: BLE001 - reported on the packet
-                    compute_error = compute_error or (
-                        f"compute: {type(exc).__name__}: {exc}"
-                    )
+                    # The zeroing is applied *here*, between two frames, and only
+                    # then reported — the same confirmation rule as the
+                    # calibration.
+                    self._consume_pending_reset(sensor, frame_id, frame_mono)
+                except Exception as exc:  # noqa: BLE001
+                    message = f"reset failed: {type(exc).__name__}: {exc}"
+                    if self._on_error is not None and session == self._session_id:
+                        self._on_error(message)
+                # The same chain the offline depth replay runs. The depth it
+                # reads is the packet's depth, so the channel is read once.
+                chain_depth, chain_error = process_chain(sensor)
+                compute_error = compute_error or chain_error
                 t1 = time.perf_counter()
                 frame_times.append(t1)
                 cutoff = t1 - 1.0
@@ -683,7 +695,7 @@ class SdkAcquisitionWorker(RecordSinkMixin):
                     stale_forces=bool(compute_error),
                     raw_available=bool(forces.get("raw_available", True)),
                     flow=forces["flow"],
-                    depth=forces.get("depth"),
+                    depth=forces.get("depth") if forces.get("depth") is not None else chain_depth,
                 )
                 with self._lock:
                     self._latest = packet

shared/src/orisys_app_shared/force/calibration_io.py

@@ -8,10 +8,10 @@ from __future__ import annotations
 
 import math
 from copy import deepcopy
-from dataclasses import asdict, dataclass, field
+from dataclasses import asdict, dataclass, field, replace
 from datetime import datetime
 from pathlib import Path
-from typing import Any, Literal, Mapping
+from typing import Any, Iterable, Literal, Mapping
 
 from orisys.calibration import (
     CalSample as SdkCalSample,
@@ -95,8 +95,6 @@ def predict_channel(cal, channel: str, raw: float) -> float:
     triple = [0.0, 0.0, 0.0]
     triple[_CONVERT_SLOT[channel]] = float(raw)
     return float(cal.convert(triple[0], triple[1], triple[2])[_CONVERT_SLOT[channel]])
-
-
 @dataclass
 class CalSessionStore:
     """In-memory calibration points for FN / FX / FY."""
@@ -176,12 +174,8 @@ def meta_from_sensor_json(path: str | Path | None) -> dict[str, Any]:
     }
 
 
-def build_force_calibration_v2(
-    store: CalSessionStore,
-    *,
-    meta: dict[str, Any] | None = None,
-) -> tuple[dict[str, Any], list[str]]:
-    """Build schema v2 JSON. Returns (document, skipped_channel_warnings)."""
+def _meta_blocks(meta: dict[str, Any] | None) -> tuple[dict, dict, dict, str]:
+    """Split a flat export-meta dict into the SDK's identity/pipeline/extra."""
     meta = dict(meta or {})
     identity = {
         "sensor_type": meta.get("sensor_type", ""),
@@ -194,9 +188,9 @@ def build_force_calibration_v2(
         "config_ref": meta.get("config_ref"),
     }
     extra = {
-        k: v
-        for k, v in meta.items()
-        if k
+        key: value
+        for key, value in meta.items()
+        if key
         not in (
             "sensor_type",
             "sn",
@@ -207,30 +201,255 @@ def build_force_calibration_v2(
             "samples",
         )
     }
+    return identity, pipeline, extra, str(meta.get("origin") or "user")
+
+
+def aggregate_samples(rows: Iterable[CalSample]) -> list[CalSample]:
+    """n-weighted merge of repeated reference loads, sorted by load.
+
+    Sampling the same load twice is normal and deliberate; the model has to see
+    the combined mean and the scatter *between* the samples, not whichever
+    sample happened to be last.
+    """
+    merged: list[CalSample] = []
+    for sample in rows:
+        for index, existing in enumerate(merged):
+            if abs(existing.weight_n - sample.weight_n) <= WEIGHT_EPS:
+                merged[index] = merge_samples(existing, sample)
+                break
+        else:
+            merged.append(sample)
+    merged.sort(key=lambda item: item.weight_n)
+    return merged
+
+
+def training_samples(points) -> tuple[dict[str, list[CalSample]], list, list]:
+    """Split points into the training build and the held-out validation set.
+
+    This is the **single** filter: a point that is excluded, invalid or has no
+    reference load never reaches a model, and a validation point is only ever
+    measured against one. Callers must not re-implement it.
+    """
+    training: list = []
+    validation: list = []
+    for point in points or ():
+        if bool(getattr(point, "excluded", False)):
+            continue
+        try:
+            load = float(point.reference_load_n)
+            raw_mean = float(point.loaded_stats().mean)
+        except (AttributeError, TypeError, ValueError):
+            continue
+        if not math.isfinite(load) or not math.isfinite(raw_mean):
+            continue
+        if bool(getattr(point, "is_validation", False)):
+            validation.append(point)
+        else:
+            training.append(point)
+
+    by_channel: dict[str, list[CalSample]] = {key: [] for key in CHANNEL_KEYS}
+    for point in training:
+        stats = point.loaded_stats()
+        std = float(stats.std)
+        by_channel[str(point.channel)].append(
+            CalSample(
+                weight_n=float(point.reference_load_n),
+                raw_mean=float(stats.mean),
+                n=int(stats.n or 0),
+                std=std if math.isfinite(std) else 0.0,
+            )
+        )
+    return (
+        {key: aggregate_samples(rows) for key, rows in by_channel.items()},
+        training,
+        validation,
+    )
+
+
+@dataclass(frozen=True)
+class PreparedCalibration:
+    """One preparation of a calibration session: samples -> model -> diagnostics.
+
+    The diagnostics and the exported document come from *this* object, so the
+    numbers on screen describe exactly the model that is written to disk. Two
+    separate preparations (one for the panel, one for the fit) is how the
+    displayed residual and the applied curve drifted apart in the first place.
+    """
+
+    model: Any = None
+    samples: dict[str, list[CalSample]] = field(default_factory=dict)
+    training: list = field(default_factory=list)
+    validation: list = field(default_factory=list)
+    #: ``id(point)`` -> prediction error at that point (predicted - reference).
+    residuals: dict[int, float] = field(default_factory=dict)
+    #: ``id(point)`` -> absolute error of a held-out point.
+    validation_errors: dict[int, float] = field(default_factory=dict)
+    warnings: list[str] = field(default_factory=list)
+    meta: dict = field(default_factory=dict)
+
+    @property
+    def ok(self) -> bool:
+        return self.model is not None
+
+    def channel_samples(self, key: str) -> list[CalSample]:
+        return list(self.samples.get(key) or [])
+
+    def method(self, channel: str):
+        curve = self.curve(channel)
+        return str(getattr(curve, "method", "") or "") if curve is not None else None
+
+    def breakpoints(self, channel: str) -> int:
+        curve = self.curve(channel)
+        return len(getattr(curve, "points", ()) or ()) if curve is not None else 0
+
+    def curve(self, channel: str):
+        if self.model is None:
+            return None
+        return self.model.get(channel)
+
+    def predict(self, channel: str, raw: float) -> float:
+        if self.model is None:
+            return float("nan")
+        return predict_channel(self.model, channel, raw)
+
+    def points_in_fit(self, channel: str) -> int:
+        return len(self.samples.get(channel) or [])
+
+    def validation_points(self, channel: str) -> int:
+        return sum(1 for point in self.validation if point.channel == channel)
+
+    def max_residual(self, channel: str) -> float:
+        values = [
+            abs(value)
+            for point in self.training
+            if point.channel == channel
+            for value in [self.residuals.get(id(point))]
+            if value is not None and math.isfinite(value)
+        ]
+        return max(values) if values else float("nan")
+
+    def max_validation_error(self, channel: str) -> float:
+        values = [
+            value
+            for point in self.validation
+            if point.channel == channel
+            for value in [self.validation_errors.get(id(point))]
+            if value is not None and math.isfinite(value)
+        ]
+        return max(values) if values else float("nan")
+
+    def to_document(self) -> dict[str, Any]:
+        """The schema-v2 document for this prepared model."""
+        if self.model is None:
+            raise ValueError(
+                "No complete channels to build (each needs ≥2 points). "
+                + "; ".join(self.warnings)
+            )
+        return self.model.to_dict()
+
+
+def prepare_from_samples(
+    samples: dict[str, list[CalSample]],
+    *,
+    meta: dict[str, Any] | None = None,
+) -> PreparedCalibration:
+    """Build the model and its diagnostics from prepared training samples.
+
+    Repeated loads are merged here too, so a caller that hands in raw (unmerged)
+    samples still gets the same model as one that hands in points.
+    """
+    merged = {
+        key: aggregate_samples(samples.get(key) or []) for key in CHANNEL_KEYS
+    }
+    identity, pipeline, extra, origin = _meta_blocks(meta)
+    warnings: list[str] = []
+    model = None
     try:
-        cal = sdk_build(
-            store.to_sdk_samples(),
+        model = sdk_build(
+            {key: [sample.to_sdk() for sample in rows] for key, rows in merged.items()},
             identity=identity,
             pipeline=pipeline,
-            origin=str(meta.get("origin") or "user"),
+            origin=origin,
             extra_meta=extra,
         )
-    except Exception as exc:
-        # Match previous API: raise ValueError with channel warnings when empty.
-        raise ValueError(str(exc)) from exc
+    except Exception as exc:  # noqa: BLE001 - surfaced as warnings + no model
+        warnings.append(str(exc))
+    if model is not None:
+        try:
+            warnings.extend(sdk_check(model).warnings)
+        except Exception as exc:  # noqa: BLE001
+            warnings.append(f"check failed: {exc}")
+        for key in CHANNEL_KEYS:
+            if not model.has(key):
+                warnings.append(_missing_channel_message(key, merged.get(key) or []))
+    return PreparedCalibration(
+        model=model,
+        samples=merged,
+        warnings=warnings,
+        meta=dict(meta or {}),
+    )
 
-    result = sdk_check(cal)
-    warnings = list(result.warnings)
-    for key in CHANNEL_KEYS:
-        if not cal.has(key):
-            rows = store.channel(key)
-            if rows:
-                msg = f"{key}: need at least 2 points (have {len(rows)})"
-            else:
-                msg = f"{key}: no samples"
-            if msg not in warnings:
-                warnings.append(msg)
-    return cal.to_dict(), warnings
+
+def _missing_channel_message(key: str, rows: list[CalSample]) -> str:
+    if rows:
+        return f"{key}: need at least 2 points (have {len(rows)})"
+    return f"{key}: no samples"
+
+
+def prepare_calibration(
+    points,
+    *,
+    meta: dict[str, Any] | None = None,
+) -> PreparedCalibration:
+    """The one entry point: points -> training samples -> model -> diagnostics."""
+    samples, training, validation = training_samples(points)
+    prepared = prepare_from_samples(samples, meta=meta)
+    residuals: dict[int, float] = {}
+    for point in training:
+        predicted = prepared.predict(str(point.channel), float(point.loaded_stats().mean))
+        if math.isfinite(predicted):
+            residuals[id(point)] = predicted - float(point.reference_load_n)
+    validation_errors: dict[int, float] = {}
+    for point in validation:
+        predicted = prepared.predict(str(point.channel), float(point.loaded_stats().mean))
+        if math.isfinite(predicted):
+            validation_errors[id(point)] = abs(predicted - float(point.reference_load_n))
+    return replace(
+        prepared,
+        training=list(training),
+        validation=list(validation),
+        residuals=residuals,
+        validation_errors=validation_errors,
+    )
+
+
+def build_export_model(samples: dict[str, list[CalSample]]):
+    """The SDK model for ``samples`` — the same preparation the panel uses."""
+    return prepare_from_samples(samples).model
+
+
+def build_force_calibration_v2(
+    source,
+    *,
+    meta: dict[str, Any] | None = None,
+) -> tuple[dict[str, Any], list[str]]:
+    """Serialise the prepared model. Returns (document, warnings).
+
+    ``source`` is a :class:`PreparedCalibration` (the normal path — the document
+    is exactly the model that was diagnosed) or, for older callers, a
+    :class:`CalSessionStore`, which is prepared here with ``meta``.
+    """
+    prepared = (
+        source
+        if isinstance(source, PreparedCalibration)
+        else prepare_from_samples(
+            {key: source.channel(key) for key in CHANNEL_KEYS}, meta=meta
+        )
+    )
+    if prepared.model is None:
+        # Match the previous API: raise ValueError carrying the channel warnings.
+        raise ValueError("; ".join(prepared.warnings) or "no complete channels")
+    return prepared.to_document(), list(prepared.warnings)
 
 
 def write_force_calibration(
@@ -245,15 +464,18 @@ def write_force_calibration(
 
 
 def export_force_calibration_v2(
-    store: CalSessionStore,
+    source,
     *,
     export_dir: str | Path,
     sensor_type: str,
     meta: dict[str, Any] | None = None,
     local_bundle_dir: str | Path | None = None,
 ) -> tuple[Path, Path | None, list[str]]:
-    """Write timestamped export and optional local bundle force_calibration.json."""
-    doc, warnings = build_force_calibration_v2(store, meta=meta)
+    """Write timestamped export and optional local bundle force_calibration.json.
+
+    ``source`` is the prepared calibration (preferred) or a session store.
+    """
+    doc, warnings = build_force_calibration_v2(source, meta=meta)
     if not doc.get("channels"):
         raise ValueError(
             "No complete channels to export (each needs ≥2 points). "

shared/src/orisys_app_shared/force/calibration_recorder.py

@@ -23,123 +23,39 @@ from __future__ import annotations
 
 import threading
 import time
-from dataclasses import dataclass, field
+from dataclasses import dataclass
 from datetime import datetime, timezone
-from typing import Callable, Literal, Optional
+from typing import Callable, Optional
 
 import numpy as np
 
-from orisys_app_shared.force.session import NA, SessionSample, summarise
+from orisys_app_shared.force.models import (
+    CHANNEL_KEYS,
+    MAX_TRACE_POINTS,
+    NA,
+    SAMPLE_FIELDS,
+    STATUS_CANCELLED,
+    STATUS_FEW_SAMPLES,
+    STATUS_OK,
+    AxisStats,
+    CalibrationPoint,
+    ChannelKey,
+    trace_dt,
+    trace_from,
+)
+from orisys_app_shared.force.session import SessionSample, summarise
 
-try:  # pragma: no cover - the module is a plain data definition
-    from orisys_app_shared.force.calibration_session import SAMPLE_FIELDS
-except Exception:  # noqa: BLE001 - a partial install must not break sampling
-    SAMPLE_FIELDS = (
-        "frame_id",
-        "t_rel_s",
-        "fn_raw",
-        "fx_raw",
-        "fy_raw",
-        "fn_n",
-        "fx_n",
-        "fy_n",
-        "valid",
-    )
-
-ChannelKey = Literal["fn", "fx", "fy"]
-CHANNEL_KEYS: tuple[ChannelKey, ...] = ("fn", "fx", "fy")
-
-STATUS_OK = "ok"
-STATUS_FEW_SAMPLES = "few_samples"
-STATUS_CANCELLED = "cancelled"
-
-
-@dataclass(frozen=True)
-class AxisStats:
-    """Mean / standard deviation / valid count for one axis."""
-
-    mean: float = NA
-    std: float = NA
-    n: int = 0
-
-    def as_dict(self) -> dict:
-        return {"mean": self.mean, "std": self.std, "n": self.n}
-
-
-@dataclass
-class CalibrationPoint:
-    """One recorded calibration point, with all three axes captured."""
-
-    channel: ChannelKey
-    reference_load_n: float
-    direction: str = "load"
-    round_index: int = 0
-    note: str = ""
-    started_at: str = ""
-    ended_at: str = ""
-    duration_s: float = 0.0
-    requested_duration_s: float = 0.0
-    n_total: int = 0
-    n_valid: int = 0
-    raw: dict[str, AxisStats] = field(default_factory=dict)
-    newtons: dict[str, AxisStats] = field(default_factory=dict)
-    status: str = STATUS_OK
-    #: Downsampled per-axis raw trace, so a row can be inspected after the fact.
-    trace: dict[str, list[float]] = field(default_factory=dict)
-    trace_dt_s: float = 0.0
-    #: **Audit** series: every sample of the point, in
-    #: :data:`~orisys_app_shared.force.calibration_session.SAMPLE_FIELDS` order.
-    #: The trace above is a preview and must never be the only surviving copy.
-    samples: list[tuple] = field(default_factory=list)
-    #: Diagnostics, all operator-controlled. Excluded points leave the fit;
-    #: validation points are held out of it and used to test it.
-    excluded: bool = False
-    is_validation: bool = False
-    #: Filled in by :meth:`CalibrationRecorder.fit`.
-    residual_n: float = NA
-
-    def loaded_stats(self) -> AxisStats:
-        return self.raw.get(self.channel, AxisStats())
-
-    def axis_mean(self, channel: str) -> float:
-        stats = self.raw.get(channel)
-        return stats.mean if stats is not None else NA
-
-    def state_key(self) -> str:
-        if self.excluded:
-            return "cal.state_excluded"
-        if self.is_validation:
-            return "cal.state_validation"
-        return "cal.state_ok"
-
-    def as_dict(self) -> dict:
-        return {
-            "channel": self.channel,
-            "reference_load_n": self.reference_load_n,
-            # Only the loaded axis has a known reference load; the other two are
-            # intentionally absent rather than zero.
-            "direction": self.direction,
-            "round_index": self.round_index,
-            "note": self.note,
-            "started_at": self.started_at,
-            "ended_at": self.ended_at,
-            "duration_s": self.duration_s,
-            "requested_duration_s": self.requested_duration_s,
-            "n_total": self.n_total,
-            "n_valid": self.n_valid,
-            "status": self.status,
-            "excluded": self.excluded,
-            "is_validation": self.is_validation,
-            "residual_n": self.residual_n,
-            "raw": {key: stats.as_dict() for key, stats in self.raw.items()},
-            "newtons": {key: stats.as_dict() for key, stats in self.newtons.items()},
-            # The audit series is part of the point's identity, not a UI extra:
-            # without it the only surviving copy of the samples is the
-            # downsampled trace.
-            "sample_fields": list(SAMPLE_FIELDS),
-            "n_samples": len(self.samples),
-            "samples": [list(row) for row in self.samples],
-        }
+# Re-exported: the models live in ``force.models`` so the session file format can
+# describe a point without importing the sampling state machine.
+__all__ = [
+    "AxisStats",
+    "CalibrationPoint",
+    "CalibrationRecorder",
+    "FitResult",
+    "SAMPLE_FIELDS",
+    "trace_dt",
+    "trace_from",
+]
 
 
 @dataclass(frozen=True)
@@ -177,10 +93,6 @@ class FitResult:
         }
 
 
-#: Cap on stored trace samples per axis; peaks are preserved when reducing.
-MAX_TRACE_POINTS = 400
-
-
 class CalibrationRecorder:
     """Owns the calibration sampling lifecycle."""
 
@@ -358,8 +270,8 @@ class CalibrationRecorder:
             raw={"fn": axis_stats(2), "fx": axis_stats(3), "fy": axis_stats(4)},
             newtons={"fn": axis_stats(5), "fx": axis_stats(6), "fy": axis_stats(7)},
             status=STATUS_OK if valid_block.shape[0] >= 2 else STATUS_FEW_SAMPLES,
-            trace=_trace_from(valid_block),
-            trace_dt_s=_trace_dt(valid_block),
+            trace=trace_from(valid_block),
+            trace_dt_s=trace_dt(valid_block),
             # Every sample is kept, including the invalid ones: a point that was
             # mostly invalid is itself a finding. ``samples`` holds the original
             # tuples so ``frame_id`` stays an integer.
@@ -456,94 +368,71 @@ class CalibrationRecorder:
         with self._lock:
             return [point for point in self.points if point.channel == channel]
 
-    def fit(self, channel: ChannelKey) -> FitResult:
+    def prepared(self, *, meta: Optional[dict] = None):
+        """The session prepared into one model plus its diagnostics.
+
+        Everything that shows or writes a calibration goes through this, so the
+        panel, the fit summary and the exported file cannot disagree about which
+        points were used.
+        """
+        from orisys_app_shared.force.calibration_io import prepare_calibration
+
+        with self._lock:
+            points = list(self.points)
+        return prepare_calibration(points, meta=meta)
+
+    def fit(self, channel: ChannelKey, prepared=None) -> FitResult:
         """Diagnose the model that export and application would actually use.
 
         The residual and validation figures used to come from a private
         least-squares line while the exported file carried a piecewise curve, so
         the numbers on screen did not describe the calibration that would be
-        applied. Both now come from the exported curve. Excluded points are
-        dropped, validation points are held out of the build and only measured.
+        applied. Both now come from the one prepared model (see
+        :func:`~orisys_app_shared.force.calibration_io.prepare_calibration`):
+        excluded points are dropped, validation points are held out of the build
+        and only measured, and repeated loads are merged before the build.
         """
-        from orisys_app_shared.force.calibration_io import (
-            CalSample,
-            build_export_model,
-            predict_channel,
-        )
+        if prepared is None:
+            prepared = self.prepared()
 
         points = self.points_for(channel)
-        usable = [
-            point
-            for point in points
-            if not point.excluded and not point.is_validation
-            and np.isfinite(point.reference_load_n)
-            and np.isfinite(point.loaded_stats().mean)
-        ]
         for point in points:
-            point.residual_n = NA
-        if len(usable) < 2:
-            return FitResult(channel=channel, n_points=len(usable))
+            value = prepared.residuals.get(id(point))
+            point.residual_n = NA if value is None else float(value)
 
-        samples: dict[str, list[CalSample]] = {key: [] for key in CHANNEL_KEYS}
-        for point in usable:
-            samples[point.channel].append(
-                CalSample(
-                    weight_n=float(point.reference_load_n),
-                    raw_mean=float(point.loaded_stats().mean),
-                    n=int(point.loaded_stats().n),
-                    std=float(point.loaded_stats().std)
-                    if np.isfinite(point.loaded_stats().std)
-                    else 0.0,
-                )
-            )
-        try:
-            model = build_export_model(samples)
-            curve = model.get(channel)
-        except Exception:
-            # Without a buildable model there is nothing honest to report.
-            return FitResult(channel=channel, n_points=len(usable))
-        if curve is None:
-            return FitResult(channel=channel, n_points=len(usable))
+        curve = prepared.curve(channel)
+        n_points = prepared.points_in_fit(channel)
+        if curve is None or n_points < 2:
+            return FitResult(channel=channel, n_points=n_points)
 
-        residuals = []
-        for point in usable:
-            predicted = predict_channel(model, point.channel, point.loaded_stats().mean)
-            residual = predicted - float(point.reference_load_n)
-            point.residual_n = float(residual)
-            residuals.append(residual)
-
-        validation = [
-            point
-            for point in points
-            if not point.excluded and point.is_validation
-            and np.isfinite(point.reference_load_n)
-            and np.isfinite(point.loaded_stats().mean)
-        ]
-        validation_error = NA
-        if validation:
-            errors = [
-                abs(
-                    predict_channel(model, point.channel, point.loaded_stats().mean)
-                    - float(point.reference_load_n)
-                )
-                for point in validation
-            ]
-            validation_error = float(max(errors))
-
-        raws = np.asarray([point.loaded_stats().mean for point in usable], dtype=float)
-        loads = np.asarray([point.reference_load_n for point in usable], dtype=float)
+        raws = np.asarray(
+            [
+                float(point.loaded_stats().mean)
+                for point in prepared.training
+                if point.channel == channel
+            ],
+            dtype=float,
+        )
+        loads = np.asarray(
+            [
+                float(point.reference_load_n)
+                for point in prepared.training
+                if point.channel == channel
+            ],
+            dtype=float,
+        )
         linear_gain = linear_offset = NA
-        if not np.allclose(raws, raws[0]):
+        if raws.size >= 2 and not np.allclose(raws, raws[0]):
             linear_gain, linear_offset = (float(v) for v in np.polyfit(raws, loads, 1))
 
         return FitResult(
             channel=channel,
-            n_points=len(usable),
-            max_residual_n=float(np.max(np.abs(residuals))),
-            validation_error_n=validation_error,
-            validation_points=len(validation),
+            n_points=n_points,
+            max_residual_n=prepared.max_residual(channel),
+            validation_error_n=prepared.max_validation_error(channel),
+            validation_points=prepared.validation_points(channel),
             method=str(getattr(curve, "method", "") or ""),
-            breakpoints=len(getattr(curve, "points", ()) or ()),
+            breakpoints=prepared.breakpoints(channel),
             linear_gain=linear_gain,
             linear_offset=linear_offset,
         )
@@ -554,37 +443,3 @@ class CalibrationRecorder:
             self._rounds[channel] = index + 1
             return index
 
-
-def _trace_from(valid_block: np.ndarray) -> dict[str, list[float]]:
-    """Downsampled raw trace per axis, bounded by ``MAX_TRACE_POINTS``."""
-    if valid_block.size == 0:
-        return {}
-    traces: dict[str, list[float]] = {}
-    for key, column in (("fn", 2), ("fx", 3), ("fy", 4)):
-        values = valid_block[:, column]
-        values = values[np.isfinite(values)]
-        if values.size == 0:
-            continue
-        if values.size > MAX_TRACE_POINTS:
-            # Keep the shape of the trace, including its extremes.
-            edges = np.linspace(0, values.size, MAX_TRACE_POINTS + 1, dtype=int)
-            reduced = [
-                values[start:max(start + 1, stop)]
-                for start, stop in zip(edges[:-1], edges[1:])
-            ]
-            values = np.asarray(
-                [
-                    bucket[int(np.argmax(np.abs(bucket)))]
-                    for bucket in reduced
-                    if bucket.size
-                ]
-            )
-        traces[key] = [float(value) for value in values]
-    return traces
-
-
-def _trace_dt(valid_block: np.ndarray) -> float:
-    if valid_block.shape[0] < 2:
-        return 0.0
-    span = float(valid_block[-1, 1] - valid_block[0, 1])
-    return span / max(1, valid_block.shape[0] - 1)

shared/src/orisys_app_shared/force/calibration_session.py

@@ -23,23 +23,19 @@ from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any, Iterable, Optional
 
+from orisys_app_shared.force.models import (
+    SAMPLE_FIELDS,
+    AxisStats,
+    CalibrationPoint,
+    trace_dt,
+    trace_from,
+)
+
 logger = logging.getLogger(__name__)
 
 #: Format version of the calibration session document.
 SESSION_FORMAT_VERSION = "1.0"
 SESSION_REQUIRED = ("format", "format_version", "created_at", "points")
-#: Columns of each sample row inside a point, in order.
-SAMPLE_FIELDS = (
-    "frame_id",
-    "t_rel_s",
-    "fn_raw",
-    "fx_raw",
-    "fy_raw",
-    "fn_n",
-    "fx_n",
-    "fy_n",
-    "valid",
-)
 POINT_REQUIRED = ("channel", "reference_load_n")
 
 
@@ -59,6 +55,29 @@ class CalibrationSessionRecord:
     def ok(self) -> bool:
         return bool(self.points) and not self.warnings
 
+    def identity(self) -> dict:
+        """The device this session belongs to.
+
+        A calibration record is evidence about *one* device: its sensor type, its
+        serial number and the configuration it was taken with. Reopening it must
+        not restamp it with whatever device happens to be connected now.
+        """
+        return {
+            "sensor_type": str(self.meta.get("sensor_type") or ""),
+            "sn": str(self.meta.get("sn") or ""),
+            "config_ref": str(self.meta.get("config_ref") or ""),
+            "created_at": self.created_at,
+        }
+
+    def describe(self) -> str:
+        identity = self.identity()
+        parts = [identity["sensor_type"] or "?"]
+        if identity["sn"]:
+            parts.append(f"SN {identity['sn']}")
+        if identity["config_ref"]:
+            parts.append(identity["config_ref"])
+        return " / ".join(parts)
+
 
 def _jsonable(value: Any) -> Any:
     """NaN/Inf are not valid JSON; they are written as ``null``."""
@@ -147,11 +166,6 @@ def read_calibration_session(path: str | Path) -> CalibrationSessionRecord:
     if version != SESSION_FORMAT_VERSION:
         warnings.append(f"format_version {version or '?'} is not {SESSION_FORMAT_VERSION}")
 
-    from orisys_app_shared.force.calibration_recorder import (
-        CalibrationPoint,
-        AxisStats,
-    )
-
     points = []
     for index, raw in enumerate(data.get("points") or []):
         if not isinstance(raw, dict):
@@ -228,9 +242,39 @@ def _point_from_dict(raw: dict, CalibrationPoint, AxisStats):
         values = [float("nan") if value is None else value for value in row]
         rows.append(tuple(values))
     point.samples = rows
+    # The full series is the audit data, so everything a diagnostic view needs is
+    # rebuilt from it: the per-axis trace, its sample interval and, when the
+    # stored duration is missing, the real sampling span.
+    restore_diagnostics(point)
     return point
 
 
+def restore_diagnostics(point) -> None:
+    """Rebuild the three-axis trace and the sampling span from the samples."""
+    if not point.samples:
+        return
+    try:
+        import numpy as np
+    except Exception:  # noqa: BLE001 - a partial install must still reopen points
+        return
+
+    width = max(len(row) for row in point.samples)
+    padded = [tuple(row) + (float("nan"),) * (width - len(row)) for row in point.samples]
+    block = np.asarray(padded, dtype=np.float64)
+    if block.shape[1] < len(SAMPLE_FIELDS):
+        return
+    valid = block[:, 8] > 0.5
+    valid_block = block[valid]
+    if not point.trace and valid_block.size:
+        point.trace = trace_from(valid_block)
+    if not point.trace_dt_s and valid_block.size:
+        point.trace_dt_s = trace_dt(valid_block)
+    if not point.duration_s and block.shape[0] > 1:
+        span = float(block[-1, 1] - block[0, 1])
+        if span == span and span >= 0.0:
+            point.duration_s = span
+
+
 def _optional_float(value: Any) -> float:
     if value is None:
         return float("nan")

shared/src/orisys_app_shared/force/models.py

@@ -0,0 +1,170 @@
+"""The calibration data shapes, independent of sampling and of file storage.
+
+They live here so the sampling state machine and the session file format can both
+describe a calibration point without importing each other. ``calibration_recorder``
+and ``calibration_session`` re-export these names for compatibility.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Literal
+
+import numpy as np
+
+#: Stands in for "this value is not available".
+NA = float("nan")
+
+ChannelKey = Literal["fn", "fx", "fy"]
+CHANNEL_KEYS: tuple[ChannelKey, ...] = ("fn", "fx", "fy")
+
+STATUS_OK = "ok"
+STATUS_FEW_SAMPLES = "few_samples"
+STATUS_CANCELLED = "cancelled"
+
+#: Columns of one audit sample row, in order. A calibration record is only
+#: auditable if this layout is fixed: it is what a reopened point is rebuilt from.
+SAMPLE_FIELDS = (
+    "frame_id",
+    "t_rel_s",
+    "fn_raw",
+    "fx_raw",
+    "fy_raw",
+    "fn_n",
+    "fx_n",
+    "fy_n",
+    "valid",
+)
+
+#: Index of each sample column, so callers do not hard-code positions.
+SAMPLE_INDEX = {name: index for index, name in enumerate(SAMPLE_FIELDS)}
+
+#: Cap on stored trace samples per axis; peaks are preserved when reducing.
+MAX_TRACE_POINTS = 400
+
+
+@dataclass(frozen=True)
+class AxisStats:
+    """Mean / standard deviation / valid count for one axis."""
+
+    mean: float = NA
+    std: float = NA
+    n: int = 0
+
+    def as_dict(self) -> dict:
+        return {"mean": self.mean, "std": self.std, "n": self.n}
+
+
+@dataclass
+class CalibrationPoint:
+    """One recorded calibration point, with all three axes captured."""
+
+    channel: ChannelKey
+    reference_load_n: float
+    direction: str = "load"
+    round_index: int = 0
+    note: str = ""
+    started_at: str = ""
+    ended_at: str = ""
+    duration_s: float = 0.0
+    requested_duration_s: float = 0.0
+    n_total: int = 0
+    n_valid: int = 0
+    raw: dict[str, AxisStats] = field(default_factory=dict)
+    newtons: dict[str, AxisStats] = field(default_factory=dict)
+    status: str = STATUS_OK
+    #: Downsampled per-axis raw trace, so a row can be inspected after the fact.
+    trace: dict[str, list[float]] = field(default_factory=dict)
+    trace_dt_s: float = 0.0
+    #: **Audit** series: every sample of the point, in :data:`SAMPLE_FIELDS` order.
+    #: The trace above is a preview and must never be the only surviving copy.
+    samples: list[tuple] = field(default_factory=list)
+    #: Diagnostics, all operator-controlled. Excluded points leave the fit;
+    #: validation points are held out of it and used to test it.
+    excluded: bool = False
+    is_validation: bool = False
+    #: Filled in by :meth:`CalibrationRecorder.fit`.
+    residual_n: float = NA
+
+    def loaded_stats(self) -> AxisStats:
+        return self.raw.get(self.channel, AxisStats())
+
+    def axis_mean(self, channel: str) -> float:
+        stats = self.raw.get(channel)
+        return stats.mean if stats is not None else NA
+
+    def state_key(self) -> str:
+        if self.excluded:
+            return "cal.state_excluded"
+        if self.is_validation:
+            return "cal.state_validation"
+        return "cal.state_ok"
+
+    def as_dict(self) -> dict:
+        return {
+            "channel": self.channel,
+            "reference_load_n": self.reference_load_n,
+            # Only the loaded axis has a known reference load; the other two are
+            # intentionally absent rather than zero.
+            "direction": self.direction,
+            "round_index": self.round_index,
+            "note": self.note,
+            "started_at": self.started_at,
+            "ended_at": self.ended_at,
+            "duration_s": self.duration_s,
+            "requested_duration_s": self.requested_duration_s,
+            "n_total": self.n_total,
+            "n_valid": self.n_valid,
+            "status": self.status,
+            "excluded": self.excluded,
+            "is_validation": self.is_validation,
+            "residual_n": self.residual_n,
+            "raw": {key: stats.as_dict() for key, stats in self.raw.items()},
+            "newtons": {key: stats.as_dict() for key, stats in self.newtons.items()},
+            # The audit series is part of the point's identity, not a UI extra:
+            # without it the only surviving copy of the samples is the
+            # downsampled trace.
+            "sample_fields": list(SAMPLE_FIELDS),
+            "n_samples": len(self.samples),
+            "samples": [list(row) for row in self.samples],
+        }
+
+
+def trace_from(valid_block: np.ndarray) -> dict[str, list[float]]:
+    """Downsampled raw trace per axis, bounded by ``MAX_TRACE_POINTS``.
+
+    One downsampling rule, whether the point was just recorded or read back from
+    a saved session.
+    """
+    if valid_block.size == 0:
+        return {}
+    traces: dict[str, list[float]] = {}
+    for key, column in (("fn", 2), ("fx", 3), ("fy", 4)):
+        values = valid_block[:, column]
+        values = values[np.isfinite(values)]
+        if values.size == 0:
+            continue
+        if values.size > MAX_TRACE_POINTS:
+            # Keep the shape of the trace, including its extremes.
+            edges = np.linspace(0, values.size, MAX_TRACE_POINTS + 1, dtype=int)
+            reduced = [
+                values[start:max(start + 1, stop)]
+                for start, stop in zip(edges[:-1], edges[1:])
+            ]
+            values = np.asarray(
+                [
+                    bucket[int(np.argmax(np.abs(bucket)))]
+                    for bucket in reduced
+                    if bucket.size
+                ]
+            )
+        traces[key] = [float(value) for value in values]
+    return traces
+
+
+def trace_dt(valid_block: np.ndarray) -> float:
+    """The sample interval of a trace, from the block it was taken from."""
+    if valid_block.shape[0] < 2:
+        return 0.0
+    span = float(valid_block[-1, 1] - valid_block[0, 1])
+    return span / max(1, valid_block.shape[0] - 1)

shared/src/orisys_app_shared/force/session.py

@@ -29,6 +29,8 @@ from typing import Iterable, Iterator, Optional
 
 import numpy as np
 
+from orisys_app_shared.record.sample import plottable_axis_mask
+
 #: Stands in for "this value is not available / not valid".
 NA = float("nan")
 
@@ -127,14 +129,20 @@ class SessionSlice:
     def plottable(self, unit: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
         """``(times, values, mask)`` for drawing, under one shared rule.
 
-        A point is drawn only when the sample is **valid** *and* the value for the
-        requested unit is finite. Filtering on finiteness alone let a stale
-        number — left behind by a failed compute step — reappear in one view
-        after another view had already excluded it.
+        ``mask`` is per axis ``(n, 3)``: a slice is drawn only where the sample is
+        **valid** *and that axis* is finite. Filtering on finiteness alone let a
+        stale number — left behind by a failed compute step — reappear in one view
+        after another view had already excluded it, and an all-axes rule hid a
+        valid FN whenever the device had no shear calibration.
         """
         values = self.value_for_unit(unit)
-        mask = self.valid & np.isfinite(values).all(axis=1)
-        return self.times[mask], values[mask], mask
+        return self.times, values, plottable_axis_mask(values, self.valid)
+
+    def plottable_axis(self, unit: str, index: int) -> tuple[np.ndarray, np.ndarray]:
+        """``(times, values)`` for one channel, with the other axes ignored."""
+        values = self.value_for_unit(unit)
+        mask = plottable_axis_mask(values, self.valid)[:, index]
+        return self.times[mask], values[mask, index]
 
 
 @dataclass(frozen=True)

shared/src/orisys_app_shared/force/source.py

@@ -0,0 +1,228 @@
+"""One interface for reading curve data, whatever it is stored in.
+
+The live charts read a bounded in-memory session store; the replay reads a
+recording on disk. Both have to answer the same questions — what is the extent,
+what is in this interval, what does the whole session look like as an envelope —
+so the plotting code asks a *source* instead of knowing which one it has.
+
+A source never hands over more than the caller's point budget, and it always
+reports what it had to do to fit: ``decimated`` means the values are a view, not
+every sample.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol
+
+import numpy as np
+
+from orisys_app_shared.force.session import NA, SessionSlice
+
+
+@dataclass(frozen=True)
+class SourceStats:
+    """What the selection readout shows, for any source."""
+
+    duration_s: float = 0.0
+    samples: int = 0
+    peaks: tuple[float, float, float] = (NA, NA, NA)
+    means: tuple[float, float, float] = (NA, NA, NA)
+    stds: tuple[float, float, float] = (NA, NA, NA)
+
+
+@dataclass
+class SourceWindow:
+    """Rows for one interval, in the shape the charts already consume."""
+
+    times: np.ndarray
+    values: np.ndarray
+    valid: np.ndarray
+    frame_ids: np.ndarray
+    unit: str = "raw"
+    decimated: bool = False
+
+    def __len__(self) -> int:
+        return int(self.times.size)
+
+    def plottable_axis(self, index: int):
+        from orisys_app_shared.record.sample import plottable_axis_mask
+
+        mask = plottable_axis_mask(self.values, self.valid)[:, index]
+        return self.times[mask], self.values[mask, index]
+
+    def statistics(self) -> SourceStats:
+        mask = np.isfinite(self.values) & self.valid[:, None]
+        peaks: list[float] = []
+        means: list[float] = []
+        stds: list[float] = []
+        for index in range(3):
+            column = self.values[:, index][mask[:, index]]
+            if column.size == 0:
+                peaks.append(NA)
+                means.append(NA)
+                stds.append(NA)
+                continue
+            peaks.append(float(np.nanmax(column)))
+            means.append(float(np.nanmean(column)))
+            stds.append(float(np.nanstd(column)))
+        span = float(self.times[-1] - self.times[0]) if self.times.size > 1 else 0.0
+        return SourceStats(
+            duration_s=span,
+            samples=int(mask.any(axis=1).sum()),
+            peaks=tuple(peaks),
+            means=tuple(means),
+            stds=tuple(stds),
+        )
+
+
+class CurveSource(Protocol):
+    """What the charts need from wherever the data lives."""
+
+    kind: str
+
+    def extent(self) -> tuple[float, float]: ...
+
+    def window(self, t0: float, t1: float, *, max_points: int | None = None) -> SourceWindow: ...
+
+    def envelope(self, *, buckets: int = 1200) -> SourceWindow: ...
+
+    def duration_s(self) -> float: ...
+
+
+class StoreCurveSource:
+    """The live session store: already in memory, bounded by its own blocks."""
+
+    kind = "memory"
+
+    def __init__(self, store=None) -> None:
+        self.store = store
+
+    def set_store(self, store) -> None:
+        self.store = store
+
+    def extent(self) -> tuple[float, float]:
+        if self.store is None:
+            return (0.0, 0.0)
+        return self.store.extent()
+
+    def duration_s(self) -> float:
+        if self.store is None:
+            return 0.0
+        first, last = self.store.extent()
+        return max(0.0, last - first)
+
+    def _slice(self, t0: float, t1: float, max_points: int | None, unit: str) -> SessionSlice:
+        if self.store is None:
+            empty = np.empty(0)
+            return SessionSlice(
+                times=empty,
+                raw=np.empty((0, 3)),
+                newtons=np.empty((0, 3)),
+                valid=np.empty(0, dtype=bool),
+                frame_ids=np.empty(0, dtype=np.int64),
+            )
+        return self.store.visible(
+            t0, t1, max_points=max_points or 0, unit=unit
+        )
+
+    def window(
+        self,
+        t0: float,
+        t1: float,
+        *,
+        max_points: int | None = None,
+        unit: str = "raw",
+    ) -> SourceWindow:
+        slice_ = self._slice(t0, t1, max_points, unit)
+        return SourceWindow(
+            times=slice_.times,
+            values=slice_.value_for_unit(unit),
+            valid=slice_.valid,
+            frame_ids=slice_.frame_ids,
+            unit=unit,
+            decimated=bool(max_points and slice_.times.size >= int(max_points)),
+        )
+
+    def envelope(self, *, buckets: int = 1200, unit: str = "raw") -> SourceWindow:
+        """The whole session, decimated — the store's own bounded view."""
+        if self.store is None:
+            return self.window(0.0, 0.0)
+        first, last = self.store.extent()
+        return self.window(first, last, max_points=int(buckets) * 2, unit=unit)
+
+    def statistics(self, t0: float, t1: float, *, unit: str = "raw") -> SourceStats:
+        if self.store is None:
+            return SourceStats()
+        stats = self.store.statistics(t0, t1, unit=unit)
+        return SourceStats(
+            duration_s=stats.duration_s,
+            samples=stats.samples,
+            peaks=stats.peaks,
+            means=stats.means,
+            stds=stats.stds,
+        )
+
+
+class RecordingCurveSource:
+    """A recording on disk: interval reads plus a cached envelope."""
+
+    kind = "disk"
+
+    def __init__(self, reader) -> None:
+        self.reader = reader
+
+    def extent(self) -> tuple[float, float]:
+        return self.reader.index().extent()
+
+    def duration_s(self) -> float:
+        return self.reader.index().duration_s()
+
+    def window(
+        self,
+        t0: float,
+        t1: float,
+        *,
+        max_points: int | None = None,
+        unit: str = "raw",
+    ) -> SourceWindow:
+        window = self.reader.window(t0, t1, unit=unit, max_points=max_points)
+        return SourceWindow(
+            times=window.times,
+            values=window.value_for_unit(unit),
+            valid=window.valid,
+            frame_ids=window.frame_ids,
+            unit=unit,
+            decimated=window.decimated,
+        )
+
+    def envelope(self, *, buckets: int = 1200, unit: str = "raw") -> SourceWindow:
+        """The whole-session envelope, from the index's cached overview."""
+        overview = self.reader.overview(unit=unit, buckets=int(buckets))
+        keep = ~np.isnan(overview.times)
+        times = overview.times[keep]
+        if times.size == 0:
+            return self.window(0.0, 0.0, unit=unit)
+        # The envelope is drawn as a band: the low and high of each bucket. It is
+        # reported as a plain window so callers need only one shape.
+        values = np.where(
+            np.isnan(overview.highs[keep]), overview.lows[keep], overview.highs[keep]
+        )
+        return SourceWindow(
+            times=times,
+            values=values,
+            valid=np.ones(times.size, dtype=bool),
+            frame_ids=np.full(times.size, -1, dtype=np.int64),
+            unit=unit,
+            decimated=True,
+        )
+
+    def statistics(self, t0: float, t1: float, *, unit: str = "raw") -> SourceStats:
+        stats = self.reader.statistics(t0, t1, unit=unit)
+        return SourceStats(
+            duration_s=stats.duration_s,
+            samples=stats.samples,
+            peaks=stats.peaks,
+            means=stats.means,
+            stds=stats.stds,
+        )

shared/src/orisys_app_shared/force/units.py

@@ -1,46 +1,104 @@
-"""Force unit selection for display."""
+"""Force unit selection for display.
+
+The decision is made **per axis** from the standardised measurement
+(:class:`~orisys_app_shared.record.sample.RecordSample`), not from a device-level
+``force_calibrated`` flag. A device can legitimately have only its normal-force
+axis calibrated: the old rule then either showed every axis in raw (losing the
+one axis that really was in newtons) or, worse, showed raw counts in the newton
+panels. An axis without a curve is now simply **unavailable** in newton mode and
+reported as such.
+"""
 
 from __future__ import annotations
 
 from dataclasses import dataclass
 
+from orisys_app_shared.record.sample import AXES
+
+#: Index of each axis inside the triplets, matching ``AXES``.
+AXIS_INDEX: dict[str, int] = {axis: index for index, axis in enumerate(AXES)}
+
 
 @dataclass(frozen=True)
 class DisplayForces:
-    fn: float
-    fx: float
-    fy: float
+    """What the live view should show for one sample."""
+
+    #: One value per axis, in ``AXES`` order. NaN means "not available".
+    values: tuple[float, float, float]
+    #: The unit the values actually are (``N`` only when they are newtons).
     unit: str
+    #: Per axis: is this value a real measurement in ``unit``?
+    available: tuple[bool, bool, bool]
+    #: The requested unit was newton but no axis had a curve, so raw is shown.
     using_raw_fallback: bool
+    #: What the operator asked for (``N``/``raw``), for the fallback message.
+    requested_unit: str
+
+    def value(self, axis: str) -> float:
+        return self.values[AXIS_INDEX[axis]]
+
+    def is_available(self, axis: str) -> bool:
+        return self.available[AXIS_INDEX[axis]]
+
+    def unavailable_axes(self) -> tuple[str, ...]:
+        return tuple(
+            axis for axis in AXES if not self.available[AXIS_INDEX[axis]]
+        )
 
 
-def select_display_forces(
-    *,
-    units_pref: str,
-    force_calibrated: bool,
-    fn: float,
-    fx: float,
-    fy: float,
-    fn_raw: float,
-    fx_raw: float,
-    fy_raw: float,
-) -> DisplayForces:
-    """Return values for plots/labels given newton/raw preference."""
+def _finite(value: float) -> bool:
+    try:
+        return value == value and value not in (float("inf"), float("-inf"))
+    except TypeError:  # pragma: no cover - defensive
+        return False
+
+
+def select_display_forces(*, units_pref: str, sample) -> DisplayForces:
+    """Pick the values to display for one standardised measurement.
+
+    ``sample`` is a
+    :class:`~orisys_app_shared.record.sample.RecordSample`, so the same conversion
+    that feeds the CSV and the session store also decides what the live charts
+    show: there is no second interpretation of what a value means.
+    """
     want_newton = str(units_pref or "newton").lower().startswith("n")
-    if want_newton and force_calibrated:
-        return DisplayForces(fn=fn, fx=fx, fy=fy, unit="N", using_raw_fallback=False)
-    if want_newton and not force_calibrated:
+    raw = tuple(sample.raw_for(axis) for axis in AXES)
+    newton = tuple(sample.newton_for(axis) for axis in AXES)
+    requested = "N" if want_newton else "raw"
+
+    if not want_newton:
         return DisplayForces(
-            fn=fn_raw,
-            fx=fx_raw,
-            fy=fy_raw,
+            values=raw,
             unit="raw",
+            available=tuple(_finite(value) for value in raw),
+            using_raw_fallback=False,
+            requested_unit=requested,
+        )
+
+    values: list[float] = []
+    available: list[bool] = []
+    for index, _axis in enumerate(AXES):
+        if _finite(newton[index]):
+            values.append(newton[index])
+            available.append(True)
+        else:
+            # Not "zero" and not the raw count: there is no newton value at all.
+            values.append(float("nan"))
+            available.append(False)
+
+    if not any(available):
+        # Nothing is calibrated: showing raw is honest as long as it is labelled.
+        return DisplayForces(
+            values=raw,
+            unit="raw",
+            available=tuple(_finite(value) for value in raw),
             using_raw_fallback=True,
+            requested_unit=requested,
         )
     return DisplayForces(
-        fn=fn_raw,
-        fx=fx_raw,
-        fy=fy_raw,
-        unit="raw",
+        values=tuple(values),
+        unit="N",
+        available=tuple(available),
         using_raw_fallback=False,
+        requested_unit=requested,
     )

shared/src/orisys_app_shared/record/__init__.py

@@ -1,5 +1,9 @@
 """Experiment recording (video + data + metadata) and replay."""
 
+from orisys_app_shared.record.calibration_snapshot import (
+    list_calibration_snapshots,
+    store_calibration_snapshot,
+)
 from orisys_app_shared.record.depth import DepthRecomputer, DepthResult
 from orisys_app_shared.record.experiment import (
     EVENTS_HEADER,
@@ -34,5 +38,7 @@ __all__ = [
     "RecordingInfo",
     "RecordingReader",
     "format_value",
+    "list_calibration_snapshots",
     "list_recordings",
+    "store_calibration_snapshot",
 ]

shared/src/orisys_app_shared/record/calibration_snapshot.py

@@ -0,0 +1,60 @@
+"""Store the coefficients a recording was made with, next to the recording.
+
+The hash of a calibration tells you *which* coefficients were applied; it does not
+let you check what they were, and the original export file can be overwritten or
+deleted afterwards. So the recording keeps its own copy:
+
+```
+outputs/recordings/<stamp>_session<NNNN>/
+    calibration/
+        force_calibration_<hash12>.json
+```
+
+The copy is content-addressed by the digest, so applying the same calibration
+twice in one session stores one file, and two different calibrations can never
+collide.
+"""
+
+from __future__ import annotations
+
+import logging
+import shutil
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+#: Directory inside a recording that holds the coefficient snapshots.
+SNAPSHOT_DIRNAME = "calibration"
+
+
+def snapshot_name(version: str, source: Path) -> str:
+    """The snapshot's file name: content-addressed, with the original extension."""
+    digest = str(version or "").strip()
+    stem = f"force_calibration_{digest[:12]}" if digest else f"force_calibration_{source.stem}"
+    return f"{stem}{source.suffix or '.json'}"
+
+
+def store_calibration_snapshot(directory: str | Path, source: str | Path, version: str) -> str:
+    """Copy ``source`` into ``directory/calibration``. Returns the relative name.
+
+    Raises on I/O failure; the caller reports it and keeps recording.
+    """
+    base = Path(directory) / SNAPSHOT_DIRNAME
+    base.mkdir(parents=True, exist_ok=True)
+    origin = Path(source)
+    target = base / snapshot_name(version, origin)
+    if not target.is_file() or target.stat().st_size != origin.stat().st_size:
+        # Same content-addressed name means the same coefficients; a size check
+        # is enough to notice a truncated previous copy.
+        shutil.copy2(origin, target)
+    return f"{SNAPSHOT_DIRNAME}/{target.name}"
+
+
+def list_calibration_snapshots(directory: str | Path) -> list[str]:
+    """Every snapshot already stored in a recording, sorted."""
+    base = Path(directory) / SNAPSHOT_DIRNAME
+    if not base.is_dir():
+        return []
+    return sorted(
+        f"{SNAPSHOT_DIRNAME}/{path.name}" for path in base.iterdir() if path.is_file()
+    )

shared/src/orisys_app_shared/record/depth.py

@@ -25,6 +25,7 @@ from typing import Optional
 
 import numpy as np
 
+from orisys_app_shared.record.depth_backends import open_replay
 from orisys_app_shared.record.playback import RecordingReader
 
 logger = logging.getLogger(__name__)
@@ -40,25 +41,72 @@ class DepthResult:
     t_rel_s: float = 0.0
     #: What a replay could not reproduce exactly (never silently empty).
     warnings: tuple[str, ...] = ()
+    #: Which recording this answer belongs to (the replay window compares it).
+    record_id: str = ""
 
     @property
     def ok(self) -> bool:
         return self.depth is not None and not self.error
 
 
+#: The replay started at a **reset**: the algorithm's state was cleared there, so
+#: the frames since that point determine it exactly.
+STATE_EXACT = "exact"
+#: The replay is preceded by the recorded pre-roll: the intended reconstruction.
+STATE_PREROLL = "preroll"
+#: Neither: the state entering the target frame is unknown and may differ from
+#: what the live run had.
+STATE_APPROXIMATE = "approximate"
+
+
+@dataclass(frozen=True)
+class ReplayStep:
+    """One leg of a replay: read ``source`` up to ``to_frame``.
+
+    ``reset_after`` is the frame at which the algorithm's state was cleared live
+    (an operator ``zeroing``). The replay reaches that frame, then calls the
+    processor's ``reset()`` — which is what the live run did — and keeps reading.
+    """
+
+    source: Path
+    to_frame: int
+    reset_after: Optional[int] = None
+
+
+#: How many SDK reads a faithful replay may cost before it is declared
+#: unrecoverable (5 minutes at 30 fps). A replay that re-reads an hour of video
+#: to answer one cursor move is not a feature; saying so is.
+MAX_REPLAY_FRAMES = 9000
+
+
 @dataclass(frozen=True)
 class ReplayPlan:
     """Where the SDK must start reading to reproduce one frame's state."""
 
     segment: Path
-    #: Frame within ``segment`` to step to (1-based, as the SDK reads them).
-    stop_at: int
-    #: Frames to feed before ``stop_at`` to rebuild the algorithm's state.
+    #: The legs of the replay, in order. The last one ends at ``stop_at``.
+    steps: tuple[ReplayStep, ...] = ()
+    #: Frame within the final segment the caller asked for (1-based).
+    stop_at: int = 0
+    #: Frames to feed from another file before the first step.
     preroll: Path | None = None
     preroll_frames: int = 0
+    #: The ``zeroing`` frame the plan is anchored to, when there is one.
+    reset_frame_id: Optional[int] = None
+    #: ``reset`` / ``preroll`` / ``segment_start``.
+    anchor: str = "segment_start"
+    #: :data:`STATE_EXACT`, :data:`STATE_PREROLL` or :data:`STATE_APPROXIMATE`.
+    state: str = STATE_APPROXIMATE
     warnings: tuple[str, ...] = ()
     error: str = ""
 
+    @property
+    def frames_to_read(self) -> int:
+        """How many SDK reads a faithful recomputation costs."""
+        return int(self.preroll_frames) + sum(
+            max(0, int(step.to_frame)) for step in self.steps
+        )
+
 
 class DepthRecomputer:
     """Recompute depth maps for one recording, on demand."""
@@ -69,6 +117,12 @@ class DepthRecomputer:
         self._segment: Optional[Path] = None
         #: How many frames the open sensor has already consumed in ``_segment``.
         self._position = 0
+        #: The depth of ``_position``, so asking for the same frame twice is a
+        #: cache hit instead of a second (impossible) read.
+        self._last_depth: Optional[np.ndarray] = None
+        #: True when an earlier frame had to be re-read without a reset boundary
+        #: to clean up after it, so the state may not match the live run.
+        self._rewound_without_anchor = False
         self.last_error = ""
         self.last_warnings: list[str] = []
 
@@ -121,46 +175,87 @@ class DepthRecomputer:
         """Decide where replay must start so ``frame`` is reproducible.
 
         The SDK's depth is the output of a stateful chain (optical-flow tracker,
-        temporal filters), so a frame cannot be recomputed in isolation. Two
-        things make the state recoverable:
+        temporal filters), so a frame cannot be recomputed in isolation. What the
+        plan states explicitly:
 
-        * an operator **reset** clears the tracker, so replaying from the reset
-          frame reproduces the state exactly — no earlier history needed;
-        * the **pre-roll** recorded with the session holds the frames acquired
-          just before recording started, which is the only history a reset-free
-          start has.
+        * **where** to start (``start_at``, plus a pre-roll file when there is one);
+        * which **reset boundary** the replay is anchored to, if any;
+        * whether the state is actually recoverable — :data:`STATE_EXACT` (the
+          replay starts at a reset), :data:`STATE_PREROLL` (the recorded pre-roll
+          precedes it) or :data:`STATE_APPROXIMATE` (neither: the depth map is
+          still produced, and the caller is told it may differ from live).
 
-        Anything else is replayed as faithfully as possible *and reported*: the
-        caller is told the state may differ instead of being shown a depth map
-        that pretends to be the live one.
+        Finding a reset event is **not** by itself a recovered state: the reset
+        frame has to exist in the index *and* in the video, and it has to be in
+        the same segment, because the SDK cannot carry its state across a segment
+        boundary.
         """
         warnings: list[str] = []
         segment_path = self.reader.segment_path(frame.segment)
         if segment_path is None:
-            return ReplayPlan(
-                segment=Path(), stop_at=0, error="segment_missing"
-            )
+            return ReplayPlan(segment=Path(), error="segment_missing")
 
+        stop_at = max(1, int(frame.segment_frame))
         reset_frame = self.reader.last_reset_frame(frame.frame_id)
         if reset_frame is not None:
             origin = self.reader.frame_of(reset_frame)
-            if origin is not None and origin.segment == frame.segment:
-                # Exact: the tracker state at ``frame`` is fully determined by
-                # the frames since the reset in this same segment.
-                return ReplayPlan(
+            if origin is None:
+                warnings.append("reset_frame_not_indexed")
+            elif origin.dropped or not origin.in_video:
+                # The boundary frame is not in the video, so the state at that
+                # point cannot be rebuilt: replay from here is an approximation.
+                warnings.append("reset_frame_not_in_video")
+            elif origin.segment == frame.segment:
+                plan = ReplayPlan(
                     segment=segment_path,
-                    stop_at=max(1, int(frame.segment_frame)),
-                    warnings=(),
+                    steps=(
+                        ReplayStep(
+                            source=segment_path,
+                            to_frame=stop_at,
+                            reset_after=int(origin.segment_frame),
+                        ),
+                    ),
+                    stop_at=stop_at,
+                    reset_frame_id=int(reset_frame),
+                    anchor="reset",
+                    state=STATE_EXACT,
                 )
-            if origin is not None:
-                # The reset is in an earlier segment. The SDK opens one video per
-                # sensor, so the state cannot be carried across the boundary; the
-                # replay restarts at this segment and says so.
-                warnings.append("state_across_segment")
+                if plan.frames_to_read > MAX_REPLAY_FRAMES:
+                    warnings.append("state_too_far_back")
+                else:
+                    return plan
+            else:
+                # The reset is in an earlier segment. The processor survives a
+                # source swap, so the state *can* be carried: read the tail of
+                # the reset's segment, then continue in the target's.
+                reset_segment = self.reader.segment_path(origin.segment)
+                last_of_reset_segment = self._last_segment_frame(origin.segment)
+                if reset_segment is None or last_of_reset_segment is None:
+                    warnings.append("state_across_segment")
+                else:
+                    plan = ReplayPlan(
+                        segment=segment_path,
+                        steps=(
+                            ReplayStep(
+                                source=reset_segment,
+                                to_frame=int(last_of_reset_segment),
+                                reset_after=int(origin.segment_frame),
+                            ),
+                            ReplayStep(source=segment_path, to_frame=stop_at),
+                        ),
+                        stop_at=stop_at,
+                        reset_frame_id=int(reset_frame),
+                        anchor="reset",
+                        state=STATE_EXACT,
+                    )
+                    if plan.frames_to_read <= MAX_REPLAY_FRAMES:
+                        return plan
+                    # Too expensive: report it rather than freeze the window.
+                    warnings.append("state_too_far_back")
 
         start = self.reader.segment_start_frame(frame.segment)
         if start is None:
-            return ReplayPlan(segment=Path(), stop_at=0, error="frame_not_indexed")
+            return ReplayPlan(segment=Path(), error="frame_not_indexed")
 
         preroll = self.reader.preroll_segment()
         preroll_frames = self.reader.preroll_frames() if preroll is not None else 0
@@ -169,19 +264,39 @@ class DepthRecomputer:
             # one are the only history that can rebuild the tracker.
             return ReplayPlan(
                 segment=segment_path,
-                stop_at=max(1, int(frame.segment_frame)),
+                steps=(ReplayStep(source=segment_path, to_frame=stop_at),),
+                stop_at=stop_at,
                 preroll=preroll,
                 preroll_frames=int(preroll_frames),
+                anchor="preroll",
+                state=STATE_PREROLL,
                 warnings=tuple(warnings),
             )
-        if int(frame.segment_frame) > 1:
-            warnings.append("state_approximate")
+        # No reset and no pre-roll: the state entering this frame is unknown.
+        warnings.append("state_approximate")
         return ReplayPlan(
             segment=segment_path,
-            stop_at=max(1, int(frame.segment_frame)),
-            warnings=tuple(warnings),
+            steps=(
+                ReplayStep(
+                    source=segment_path,
+                    to_frame=stop_at,
+                ),
+            ),
+            stop_at=stop_at,
+            anchor="segment_start",
+            state=STATE_APPROXIMATE,
+            warnings=tuple(dict.fromkeys(warnings)),
         )
 
+    def _last_segment_frame(self, segment: int) -> Optional[int]:
+        """The highest indexed frame number inside ``segment``."""
+        frames = [
+            int(entry.segment_frame)
+            for entry in self.reader.info().frames
+            if int(entry.segment) == int(segment)
+        ]
+        return max(frames) if frames else None
+
     # ----------------------------------------------------------------- recompute
     def depth_at(self, t_rel_s: float) -> DepthResult:
         """Recompute the depth map for the frame at ``t_rel_s``."""
@@ -219,32 +334,95 @@ class DepthRecomputer:
             logger.exception("Depth recomputation failed")
             self.last_error = f"{type(exc).__name__}: {exc}"
             return DepthResult(
-                error=self.last_error, frame_id=frame.frame_id, t_rel_s=frame.t_rel_s
+                error=self.last_error,
+                frame_id=frame.frame_id,
+                t_rel_s=frame.t_rel_s,
+                warnings=plan.warnings,
+                record_id=self.record_id(),
             )
         self.last_warnings = list(plan.warnings)
+        if self._rewound_without_anchor and "state_approximate" not in self.last_warnings:
+            # The plan looked recoverable, but the read path had to restart a file
+            # without a reset boundary to clear the tracker afterwards.
+            self.last_warnings.append("state_approximate_after_rewind")
+        warnings = tuple(dict.fromkeys(self.last_warnings))
         if depth is None:
             self.last_error = "depth_not_available"
             return DepthResult(
                 error=self.last_error,
                 frame_id=frame.frame_id,
                 t_rel_s=frame.t_rel_s,
-                warnings=plan.warnings,
+                warnings=warnings,
+                record_id=self.record_id(),
             )
         self.last_error = ""
         return DepthResult(
             depth=depth,
             frame_id=frame.frame_id,
             t_rel_s=frame.t_rel_s,
-            warnings=plan.warnings,
+            warnings=warnings,
+            record_id=self.record_id(),
         )
 
+    def record_id(self) -> str:
+        """Which recording these answers belong to."""
+        return str(getattr(self.reader, "directory", "") or "")
+
+    # ------------------------------------------------------------------ playback
     def _replay(self, plan: ReplayPlan) -> Optional[np.ndarray]:
-        """Read the pre-roll (if any), then step to the planned frame."""
+        """Execute the plan's steps, keeping the processor and swapping inputs.
+
+        The **processor is kept** across a source change: the SDK's tracker lives
+        on the object, so disposing of it would dispose of the state the pre-roll
+        (or the reset boundary) just rebuilt. A reset boundary is re-applied by
+        calling ``reset()`` where the operator's ``zeroing`` happened.
+        """
+        self._rewound_without_anchor = False
         if plan.preroll is not None and plan.preroll_frames > 0:
             self._feed(plan.preroll, int(plan.preroll_frames))
-        if self._segment != plan.segment or self._position > plan.stop_at:
-            self._reopen_segment(plan.segment)
-        return self._advance_to(int(plan.stop_at))
+        depth: Optional[np.ndarray] = None
+        for step in plan.steps:
+            depth = self._run_step(plan, step)
+        return depth
+
+    def _run_step(self, plan: ReplayPlan, step: ReplayStep) -> Optional[np.ndarray]:
+        target = max(1, int(step.to_frame))
+        if self._sensor is None or self._segment != step.source:
+            if self._sensor is None or not self._swap_source(step.source):
+                self._open_segment(step.source)
+        elif self._position > target:
+            # Re-reading an earlier frame can only be done from the start of the
+            # file. Without a reset boundary to clean up after it, the tracker has
+            # now seen frames it should not have.
+            if not self._swap_source(step.source):
+                self._open_segment(step.source)
+            if plan.anchor != "reset":
+                self._rewound_without_anchor = True
+
+        last: Optional[np.ndarray] = (
+            self._last_depth if self._position == target else None
+        )
+        while self._position < target:
+            depth = self._read_one()
+            if depth is None:
+                break
+            last = depth
+            if step.reset_after is not None and self._position == int(step.reset_after):
+                # The operator's zeroing: the live run cleared the tracker here.
+                if not self._reset_processor():
+                    self.last_warnings.append("reset_not_applied")
+        return last
+
+    def _reset_processor(self) -> bool:
+        replay = self._sensor
+        reset = getattr(replay, "reset", None)
+        if not callable(reset):
+            return False
+        try:
+            return bool(reset())
+        except Exception:  # noqa: BLE001 - reported as a replay warning
+            logger.exception("Could not re-apply the reset boundary")
+            return False
 
     def _depth_from_segment(
         self, segment: Path, segment_frame: int
@@ -253,28 +431,55 @@ class DepthRecomputer:
 
         The SDK's video source only advances, so stepping costs one read per
         frame — and the position is tracked, so asking for frame 2 and then
-        frame 3 reads two frames and then one, instead of two more.
+        frame 3 reads two frames and then one, instead of two more. Asking twice
+        for the *same* frame is answered from the cached depth of that frame.
         """
         target = max(1, int(segment_frame))
-        if self._segment != segment or self._position > target:
-            self._reopen_segment(segment)
-        return self._advance_to(target)
+        if self._segment != segment:
+            self._open_segment(segment)
+        return self._run_step(
+            ReplayPlan(segment=Path(segment), stop_at=target),
+            ReplayStep(source=Path(segment), to_frame=target),
+        )
 
     # --------------------------------------------------------------------- video
     def sensor_type(self) -> str:
         device = self.reader.session().get("device") or {}
         return str(device.get("sensor_type") or "box")
 
-    def _reopen_segment(self, segment: Path) -> None:
+    def _open_segment(self, segment: Path) -> None:
+        """Open a segment on a **new** processor (the previous one is closed)."""
         self._close()
-        if self.sensor_type() == "finger":
-            self._sensor = _FingerReplay(
-                segment, self.resolved_config(), self._finger_export_dir()
-            )
-        else:
-            self._sensor = _BoxReplay(segment, self.resolved_config())
+        self._sensor = open_replay(
+            self.sensor_type(),
+            Path(segment),
+            self.resolved_config(),
+            self._finger_export_dir(),
+        )
         self._segment = Path(segment)
         self._position = 0
+        self._last_depth = None
+
+    def _swap_source(self, segment: Path) -> bool:
+        """Replace only the video source of the open processor.
+
+        Returns False when the backend cannot do it, in which case the caller
+        falls back to a full reopen (and the algorithm state is lost, which the
+        plan already reports as approximate).
+        """
+        replay = self._sensor
+        if replay is None:
+            return False
+        try:
+            if not replay.set_source(Path(segment)):
+                return False
+        except Exception:  # noqa: BLE001 - reported by the plan's warnings
+            logger.exception("Could not swap the replayed video source")
+            return False
+        self._segment = Path(segment)
+        self._position = 0
+        self._last_depth = None
+        return True
 
     def _finger_export_dir(self):
         """The finger pipeline assets, resolved exactly as the live run did."""
@@ -292,29 +497,17 @@ class DepthRecomputer:
         """Consume ``frames`` frames from another file to rebuild state."""
         if frames <= 0:
             return
-        if self._sensor is None or self._segment != segment:
-            self._reopen_segment(segment)
+        if self._sensor is None:
+            self._open_segment(segment)
+        elif self._segment != segment:
+            if not self._swap_source(segment):
+                self._open_segment(segment)
         for _ in range(int(frames)):
             if self._read_one() is None:
                 break
 
-    def _advance_to(self, target: int) -> Optional[np.ndarray]:
-        last_depth = None
-        while self._position < int(target):
-            depth = self._read_one()
-            if depth is None:
-                break
-            last_depth = depth
-        return last_depth
-
     def _read_one(self) -> Optional[np.ndarray]:
-        """Read one frame through the **live** processing chain.
-
-        ``get_img``/``read_source`` alone is not the live pipeline: the SDK's
-        depth map is produced by ``compute_deformation`` + ``compute_contact``
-        (and, for the finger, the unrolling step), so skipping them would replay
-        a different algorithm than the one that produced the recording.
-        """
+        """Read one frame through the **live** processing chain (see ``process_chain``)."""
         replay = self._sensor
         if replay is None:
             return None
@@ -322,12 +515,14 @@ class DepthRecomputer:
         if depth is None:
             return None
         self._position += 1
+        self._last_depth = depth
         return depth
 
     def _close(self) -> None:
         replay, self._sensor = self._sensor, None
         self._segment = None
         self._position = 0
+        self._last_depth = None
         if replay is None:
             return
         try:
@@ -343,97 +538,3 @@ class DepthRecomputer:
 
     def __exit__(self, *_exc) -> None:
         self.close()
-
-
-class _BoxReplay:
-    """The box sensor's live chain, opened on one video file."""
-
-    def __init__(self, segment: Path, config_ref: str) -> None:
-        import orisys
-
-        self._sensor = orisys.Sensor(
-            str(segment),
-            config_name=config_ref,
-            verbose=False,
-            backend="auto",
-        )
-
-    def read(self) -> Optional[np.ndarray]:
-        image = self._sensor.get_img()
-        if image is None:
-            return None
-        compute_deformation = getattr(self._sensor, "compute_deformation", None)
-        compute_contact = getattr(self._sensor, "compute_contact", None)
-        if callable(compute_deformation) and callable(compute_contact):
-            # Same calls and arguments as the acquisition loop.
-            compute_deformation(decompose=True, check_motion=False)
-            compute_contact()
-        else:
-            logger.warning(
-                "Sensor %r has no processing chain; replay depth is not the live algorithm",
-                type(self._sensor).__name__,
-            )
-        info = self._sensor.info
-        if not hasattr(info, "DEPTH"):
-            return None
-        value = self._sensor.read_info(info.DEPTH)
-        return None if value is None else np.asarray(value)
-
-    def close(self) -> None:
-        self._sensor.disconnect()
-
-
-class _FingerReplay:
-    """The finger's live chain: unroll the source frame, then process the tactile."""
-
-    def __init__(self, segment: Path, config_ref: str, export_dir) -> None:
-        from orisys.finger import FingerRuntime
-
-        from orisys_app_shared.acquisition.constants import (
-            FORCE_SAMPLE_RATE_HZ,
-            LOWPASS_CUTOFF_HZ,
-        )
-
-        kwargs: dict = {
-            "config": config_ref,
-            "video": str(segment),
-            "verbose": False,
-        }
-        if export_dir is not None:
-            kwargs["export_dir"] = str(export_dir)
-        self._runtime = FingerRuntime.open(**kwargs)
-        self._runtime.warmup(color=False)
-        tactile = getattr(self._runtime, "tactile", None)
-        if tactile is not None and hasattr(tactile, "configure"):
-            try:
-                # The same filter settings the acquisition loop applies.
-                tactile.configure(
-                    lowpass_cutoff_hz=LOWPASS_CUTOFF_HZ,
-                    lowpass_sample_rate_hz=FORCE_SAMPLE_RATE_HZ,
-                )
-            except Exception:  # noqa: BLE001 - the defaults still work
-                logger.exception("Could not configure the replayed tactile sensor")
-
-    def read(self) -> Optional[np.ndarray]:
-        ok, source_frame = self._runtime.read_source()
-        if not ok or source_frame is None:
-            return None
-        unrolled = self._runtime.process_frame(source_frame, color=False)
-        if unrolled is None:
-            return None
-        tactile = self._runtime.tactile
-        compute_deformation = getattr(tactile, "compute_deformation", None)
-        compute_contact = getattr(tactile, "compute_contact", None)
-        if callable(compute_deformation) and callable(compute_contact):
-            compute_deformation(decompose=True, check_motion=False)
-            compute_contact()
-        else:
-            logger.warning("Finger tactile sensor has no processing chain")
-        info = tactile.info
-        if not hasattr(info, "DEPTH"):
-            return None
-        value = tactile.read_info(info.DEPTH)
-        return None if value is None else np.asarray(value)
-
-    def close(self) -> None:
-        self._runtime.close()

shared/src/orisys_app_shared/record/depth_backends.py

@@ -0,0 +1,194 @@
+"""Replay backends: the live processing chain, pointed at a video file.
+
+Each backend keeps **one** processor for its lifetime. The tracker state lives on
+that object, so a replay must be able to change the *input source* without
+throwing the processor away — that is what makes a pre-roll, or a zeroing that
+happened in an earlier segment, reproducible at all.
+
+The processing itself is not duplicated here: the reads go through
+``acquisition.sdk_worker.process_chain``, the same function the live workers run.
+"""
+
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+from typing import Optional
+
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+
+class BoxReplay:
+    """The box sensor's live chain, opened on one video file."""
+
+    def __init__(self, segment: Path, config_ref: str) -> None:
+        import orisys
+
+        self._config_ref = config_ref
+        self._sensor = orisys.Sensor(
+            str(segment),
+            config_name=config_ref,
+            verbose=False,
+            backend="auto",
+        )
+
+    def set_source(self, segment: Path) -> bool:
+        """Point the same processor at another file.
+
+        The tracker lives on the ``Sensor`` object, so re-opening the sensor would
+        throw away the state the pre-roll just rebuilt. Only the capture is
+        replaced, and the tracker's reference frame is preserved.
+        """
+        sensor = self._sensor
+        open_source = getattr(sensor, "_open_video_source", None)
+        if not callable(open_source):
+            return False
+        initial = getattr(sensor, "initial_img", None)
+        try:
+            import orisys
+
+            sensor.vid_src = orisys.capture.normalize_source(str(segment))
+        except Exception:  # noqa: BLE001 - fall back to assigning the path
+            sensor.vid_src = str(segment)
+        sensor._is_live_camera = False
+        open_source(False, None)
+        if initial is not None:
+            # The tracker's reference frame belongs to the session, not to the
+            # file: replacing it would silently reset what "no contact" means.
+            sensor.initial_img = initial
+        return True
+
+    def read(self) -> Optional[np.ndarray]:
+        from orisys_app_shared.acquisition.sdk_worker import process_chain
+
+        image = self._sensor.get_img()
+        if image is None:
+            return None
+        # The same function the acquisition loop runs.
+        depth, error = process_chain(self._sensor)
+        if error:
+            logger.warning("Replay processing chain reported: %s", error)
+        return depth
+
+    def reset(self) -> bool:
+        """Clear the tracker, exactly as the operator's zeroing did live."""
+        reset = getattr(self._sensor, "reset", None)
+        if not callable(reset):
+            return False
+        reset()
+        return True
+
+    def close(self) -> None:
+        self._sensor.disconnect()
+
+
+class FingerReplay:
+    """The finger's live chain: unroll the source frame, then process the tactile."""
+
+    def __init__(self, segment: Path, config_ref: str, export_dir) -> None:
+        from orisys.finger import FingerRuntime
+
+        from orisys_app_shared.acquisition.constants import (
+            FORCE_SAMPLE_RATE_HZ,
+            LOWPASS_CUTOFF_HZ,
+        )
+
+        self._config_ref = config_ref
+        self._export_dir = export_dir
+        kwargs: dict = {
+            "config": config_ref,
+            "video": str(segment),
+            "verbose": False,
+        }
+        if export_dir is not None:
+            kwargs["export_dir"] = str(export_dir)
+        self._runtime = FingerRuntime.open(**kwargs)
+        self._runtime.warmup(color=False)
+        tactile = getattr(self._runtime, "tactile", None)
+        if tactile is not None and hasattr(tactile, "configure"):
+            try:
+                # The same filter settings the acquisition loop applies.
+                tactile.configure(
+                    lowpass_cutoff_hz=LOWPASS_CUTOFF_HZ,
+                    lowpass_sample_rate_hz=FORCE_SAMPLE_RATE_HZ,
+                )
+            except Exception:  # noqa: BLE001 - the defaults still work
+                logger.exception("Could not configure the replayed tactile sensor")
+
+    def set_source(self, segment: Path) -> bool:
+        """Point the same runtime at another video, keeping the tactile pipeline."""
+        try:
+            from orisys.capture import (
+                CaptureConfig,
+                ReconnectPolicy,
+                is_live_camera,
+                normalize_source,
+                open_capture,
+            )
+
+            runtime = self._runtime
+            video = normalize_source(str(segment))
+            camera = dict(getattr(runtime.geometry, "config", {}).get("camera", {}))
+            capture_config = CaptureConfig.from_camera_dict(
+                video,
+                camera,
+                reconnect=ReconnectPolicy(enabled=False),
+                verbose=False,
+                background=False,
+            )
+            capture_source = open_capture(capture_config, start_background=False)
+            if not capture_source.is_open():
+                return False
+            if is_live_camera(video):
+                capture_source.start_background()
+            previous = runtime.capture_source
+            runtime.capture_source = capture_source
+            runtime.capture = capture_source.cap
+            runtime.capture_worker = capture_source
+            if previous is not None:
+                try:
+                    previous.close()
+                except Exception:  # noqa: BLE001
+                    logger.exception("Could not close the previous finger source")
+            return True
+        except Exception:  # noqa: BLE001 - reported as an approximate replay
+            logger.exception("Could not swap the finger video source")
+            return False
+
+    def read(self) -> Optional[np.ndarray]:
+        from orisys_app_shared.acquisition.sdk_worker import process_chain
+
+        ok, source_frame = self._runtime.read_source()
+        if not ok or source_frame is None:
+            return None
+        unrolled = self._runtime.process_frame(source_frame, color=False)
+        if unrolled is None:
+            return None
+        depth, error = process_chain(self._runtime.tactile)
+        if error:
+            logger.warning("Replay processing chain reported: %s", error)
+        return depth
+
+    def reset(self) -> bool:
+        """Clear the tactile tracker, as the operator's zeroing did live."""
+        tactile = getattr(self._runtime, "tactile", None)
+        target = tactile if callable(getattr(tactile, "reset", None)) else None
+        if target is None:
+            sensor = getattr(tactile, "raw_sensor", None)
+            target = sensor if callable(getattr(sensor, "reset", None)) else None
+        if target is None:
+            return False
+        target.reset()
+        return True
+
+    def close(self) -> None:
+        self._runtime.close()
+
+
+def open_replay(sensor_type: str, segment: Path, config_ref: str, export_dir=None):
+    """The backend for a recorded sensor type."""
+    if str(sensor_type) == "finger":
+        return FingerReplay(segment, config_ref, export_dir)
+    return BoxReplay(segment, config_ref)

shared/src/orisys_app_shared/record/experiment.py

@@ -41,6 +41,7 @@ import cv2
 import numpy as np
 
 from orisys_app_shared.record.sample import RecordSample, to_record_sample
+from orisys_app_shared.record.video_writer import VideoSegments
 
 logger = logging.getLogger(__name__)
 
@@ -87,7 +88,8 @@ MAX_CONSECUTIVE_ERRORS = 5
 ALARM_EVERY_N_DROPS = 1000
 #: How many lost-row gaps may be remembered for the frame index.
 GAP_BACKLOG = 4096
-#: How many data rows may be parked while the item queue is full of video.
+#: Hard cap on the ordered stream. Past this the producer has to give up a row,
+#: and it says so by writing a gap marker instead.
 ROW_BACKLOG = 8192
 
 
@@ -115,6 +117,9 @@ class RecordConfig:
     fps: float = 30.0
     codec: str = "mp4v"
     segment_seconds: float = 300.0
+    #: Stream depth at which **video** starts being shed. Data rows are not
+    #: dropped by this limit: a row is tiny, and losing one because the encoder
+    #: fell behind is exactly the defect this bound must not reintroduce.
     max_queue: int = 2048
     #: Byte budget for the video queue. The item count alone is not a bound: at
     #: 1920x1080 BGR, 2048 queued frame copies would be ~11.9 GiB.
@@ -206,14 +211,16 @@ class ExperimentRecorder:
 
     def __init__(self, config: Optional[RecordConfig] = None) -> None:
         self.config = config or RecordConfig()
-        self._queue: queue.Queue = queue.Queue(maxsize=max(16, self.config.max_queue))
-        #: Bounded side channel for frames whose data row was lost to a full
-        #: queue. It is tiny, so it never grows with the recording.
+        #: One ordered stream for rows and events. It is **unbounded** on count:
+        #: the producer checks ``ROW_BACKLOG``/``max_queue`` itself, so a slow
+        #: encoder can no longer make the *queue* quietly delete a data row, and
+        #: rows are written in the order they were acquired.
+        self._queue: queue.Queue = queue.Queue()
+        #: Stream positions of frames whose data row was lost. Kept out of the
+        #: stream because there is no room for them there; flushed in order.
         self._gaps: deque = deque(maxlen=GAP_BACKLOG)
-        #: Rows that did not fit in the item queue because it was full of video.
-        #: They are tiny, so a small backlog covers any realistic stall.
-        self._row_backlog: deque = deque(maxlen=ROW_BACKLOG)
         self._queued_bytes = 0
+        self._seq = 0
         self._thread: Optional[threading.Thread] = None
         self._stop = threading.Event()
         self._lock = threading.Lock()
@@ -242,6 +249,8 @@ class ExperimentRecorder:
         self._preroll_segment: Optional[str] = None
         #: Problems the writer found while closing, carried into the report.
         self._writer_problems: list[str] = []
+        #: Coefficient files stored inside the recording directory.
+        self._calibration_snapshots: list[dict] = []
 
     # ------------------------------------------------------------------ state
     @property
@@ -318,7 +327,20 @@ class ExperimentRecorder:
             self._state = STATE_STARTING
         stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
         directory = Path(root) / f"{stamp}_session{session_id:04d}"
-        directory.mkdir(parents=True, exist_ok=True)
+        try:
+            directory.mkdir(parents=True, exist_ok=True)
+        except Exception as exc:  # noqa: BLE001 - reported to the caller
+            # A directory that cannot be created is a *terminal* failure, not a
+            # stuck ``starting``: the operator can fix the path and press record
+            # again, and the recorder must be willing to try.
+            with self._lock:
+                self._stats.failed = True
+                self._stats.last_error = f"could not create {directory}: {exc}"
+                self._state = STATE_FAILED
+                self._accepting = False
+                self._directory = None
+            self._alarm(f"could not create the recording directory: {exc}")
+            raise
         with self._lock:
             self._directory = directory
             self._stats = RecordStats()
@@ -335,9 +357,10 @@ class ExperimentRecorder:
             self._preroll_frames = 0
             self._preroll_segment = None
             self._writer_problems = []
+            self._calibration_snapshots = []
             # Stale bookkeeping from a previous run must not leak into this one.
             self._gaps.clear()
-            self._row_backlog.clear()
+            self._seq = 0
             self._queued_bytes = 0
             # Accept immediately: see ``_accepts_packets``.
             self._accepting = True
@@ -370,26 +393,26 @@ class ExperimentRecorder:
             self._state = STATE_STOPPING
             self._accepting = False
         self._stop.set()
-        try:
-            self._queue.put_nowait(("stop", None))
-        except queue.Full:  # pragma: no cover - writer will drain and see the flag
-            pass
+        # The stop marker is an ordinary item in the stream: everything acquired
+        # before it is written before it, and the writer stops after it.
+        self._enqueue(("stop", self._next_seq(), None))
         thread = self._thread
         if wait and thread is not None and thread.is_alive():
             thread.join(timeout=timeout)
         if thread is not None and thread.is_alive():
-            # Still writing: do not claim the experiment was recorded.
-            report = RecordReport(
+            # Either we were asked not to wait, or the writer missed the deadline.
+            # A finished recording is only ever reported by the thread that closed
+            # the files, so this is explicitly *not* a completion — and it is not
+            # cached as the report either, so a later stop() still sees the real
+            # one (``recorder.report`` stays None until then).
+            return RecordReport(
                 directory=self._directory,
                 stats=self.stats(),
                 problems=[
-                    "the recording writer did not stop within "
-                    f"{timeout:g}s; the files are not closed yet"
+                    "the recording writer has not finished closing the files yet"
                 ],
                 completed=False,
             )
-            self._report = report
-            return report
         self._thread = None
         return self._report or self._finalise()
 
@@ -398,13 +421,34 @@ class ExperimentRecorder:
 
         A session can outlive the coefficients it started with, so the version is
         per packet, not per session: the acquisition thread stamps every packet
-        with the calibration that was loaded when it was acquired.
+        with the calibration that was loaded when it was acquired. An empty
+        version is meaningful — it says the device has no calibration loaded.
         """
         with self._lock:
             self._calibration_version = str(version or "")
         if self.is_recording:
             self.log_event("calibration_version", self._calibration_version)
 
+    def note_calibration_snapshot(
+        self,
+        *,
+        version: str,
+        name: str,
+        frame_id: int = -1,
+        mono_t: Optional[float] = None,
+    ) -> None:
+        """Remember a coefficient file stored inside the recording directory."""
+        entry = {
+            "version": str(version or ""),
+            "file": str(name or ""),
+            "frame_id": int(frame_id),
+            "mono_t": None if mono_t is None else round(float(mono_t), 6),
+            "at": datetime.now(timezone.utc).isoformat(timespec="milliseconds"),
+        }
+        with self._lock:
+            self._calibration_snapshots.append(entry)
+        logger.info("Calibration snapshot stored: %s", name)
+
     def log_event(
         self,
         kind: str,
@@ -433,21 +477,29 @@ class ExperimentRecorder:
             "kind": str(kind),
             "detail": str(detail or ""),
         }
-        try:
-            self._queue.put_nowait(("event", event))
-        except queue.Full:
+        # Events ride the same ordered stream as the rows, so an operator action
+        # cannot overtake the samples it was taken between.
+        if self._queue.qsize() >= ROW_BACKLOG:
+            # The stream is saturated: an event is the cheapest thing to give up,
+            # but the loss is counted rather than silent.
             self._note_drop("event")
+            return
+        self._enqueue(("event", self._next_seq(), event))
 
     # ----------------------------------------------------------- sink interface
     def on_packet(self, packet) -> None:
         """Called on the acquisition thread: enqueue, never do I/O here.
 
-        A data row is small and a frame is large, so video is what gives way.
-        When the byte budget is exhausted the **frame** is shed, and when even
-        the item queue is full the frame is shed too and the row is parked in a
-        small backlog. Losing the row because the video could not keep up is what
-        made a video overflow look like a video-only loss while CSV rows quietly
-        disappeared.
+        A data row is small and a frame is large, so video is what gives way —
+        and the two are decided **separately**:
+
+        * the rows go into one ordered, unbounded stream (bounded only by
+          ``ROW_BACKLOG``, at which point the loss is counted and marked);
+        * the frames are bounded by a byte budget, and a frame that would exceed
+          it is shed *before* it is queued, so nothing has to be rolled back.
+
+        The stream being ordered is what makes the files ordered: rows are written
+        in the order they were acquired, no matter how far behind the encoder is.
         """
         if not self._accepts_packets():
             with self._lock:
@@ -462,36 +514,53 @@ class ExperimentRecorder:
         with self._lock:
             self._last_t = sample.mono_t
 
+        # Depth is only consulted to decide the *video*: a row is never refused
+        # because the encoder is behind, only when the stream itself is full.
+        depth = self._queue.qsize()
+        if depth >= ROW_BACKLOG:
+            self._note_gap(sample)
+            return
+
         frame = None
         if self.config.video or self.config.lossless_frames:
             candidate = self._capture_frame(packet)
-            size = 0 if candidate is None else int(candidate.nbytes)
-            with self._lock:
-                affordable = (self._queued_bytes + size) <= max(
-                    1024, int(self.config.max_queue_bytes)
-                )
-            if candidate is None:
-                pass
-            elif affordable:
-                frame = candidate
-                with self._lock:
-                    self._queued_bytes += size
-            else:
-                self._shed_video("video byte budget reached")
+            if candidate is not None:
+                size = int(candidate.nbytes)
+                if depth >= max(16, int(self.config.max_queue)):
+                    self._shed_video("record queue overflow")
+                elif not self._affordable(size):
+                    self._shed_video("video byte budget reached")
+                else:
+                    frame = candidate
+                    # Reserved here and released exactly once, by the writer that
+                    # consumes it (see ``_handle_frame``).
+                    with self._lock:
+                        self._queued_bytes += size
+        self._enqueue(("frame", self._next_seq(), sample, frame))
+
+    def _next_seq(self) -> int:
+        with self._lock:
+            self._seq += 1
+            return self._seq
+
+    def _affordable(self, size: int) -> bool:
+        with self._lock:
+            return (self._queued_bytes + int(size)) <= max(
+                1024, int(self.config.max_queue_bytes)
+            )
+
+    def _enqueue(self, item) -> bool:
+        """Append to the ordered stream. False when the stream is full.
+
+        The queue itself is unbounded — its depth is bounded by ``ROW_BACKLOG``
+        and checked by the callers — so a slow writer can never make the producer
+        drop data behind its back.
+        """
         try:
-            self._queue.put_nowait(("frame", (sample, frame)))
-        except queue.Full:
-            if frame is not None:
-                # The queue is full of video: the row is tiny and must survive.
-                frame = None
-                self._shed_video("record queue overflow")
-            with self._lock:
-                room = len(self._row_backlog) < self._row_backlog.maxlen
-                if room:
-                    self._row_backlog.append(sample)
-            if not room:
-                # Even the row backlog is full: the hole is at least marked.
-                self._note_gap(sample)
+            self._queue.put_nowait(item)
+            return True
+        except queue.Full:  # pragma: no cover - the queue is unbounded
+            return False
 
     def _shed_video(self, reason: str) -> None:
         with self._lock:
@@ -505,14 +574,26 @@ class ExperimentRecorder:
             )
 
     def _note_gap(self, sample) -> None:
-        """Remember a frame whose row was lost, so the replay can show the hole."""
+        """Remember a frame whose row was lost, so the replay can show the hole.
+
+        The gap carries the stream position it happened at. That is what lets the
+        writer put the hole in the *right place* in ``frames.csv`` instead of
+        appending it after newer frames (which would make the index unsorted).
+        """
         with self._lock:
             self._stats.dropped_data += 1
             if len(self._gaps) >= self._gaps.maxlen:
                 # The gap backlog is bounded, so a long overflow loses the marker.
                 self._stats.unrecorded_gaps += 1
                 return
-            self._gaps.append((sample.frame_id, sample.mono_t, sample.wall_t))
+            self._gaps.append(
+                (
+                    self._seq + 1,
+                    sample.frame_id,
+                    sample.mono_t,
+                    sample.wall_t,
+                )
+            )
 
     @staticmethod
     def _capture_frame(packet):
@@ -574,6 +655,7 @@ class ExperimentRecorder:
                 session_id=self._session_id,
                 calibration_id=self._calibration_id,
                 preroll=self._preroll,
+                on_problem=self._record_failure,
             )
             # Written once the pre-roll file exists, so session.json never names
             # a file that was not created.
@@ -583,26 +665,40 @@ class ExperimentRecorder:
                 self._preroll = []
             self._write_session_json(final=False)
             with self._lock:
-                self._state = STATE_RECORDING
-                self._accepting = True
+                # A stop *or a failure* that arrived while the files were opening
+                # must not be undone here: accepting packets again after ``stop``
+                # would make a finished recording look alive, and overwriting
+                # ``failed`` with ``recording`` would hide a dead writer. The
+                # stream is still drained — those rows were accepted.
+                if self._stop.is_set() or self._state in (
+                    STATE_STOPPING,
+                    STATE_FAILED,
+                ):
+                    self._accepting = False
+                else:
+                    self._state = STATE_RECORDING
+                    self._accepting = True
             while True:
-                # Parked rows and gaps first: a stop request must not strand the
-                # data this session already accepted.
-                self._drain_rows(writer)
-                self._drain_gaps(writer)
+                # Gaps first, but only those that happened *before* the item about
+                # to be written: the hole must appear in the frame index where it
+                # belongs, not after newer frames.
                 if self._stop.is_set() and self._queue.empty():
+                    self._flush_gaps(writer, None)
                     break
                 try:
-                    kind, payload = self._queue.get(timeout=0.2)
+                    item = self._queue.get(timeout=0.2)
                 except queue.Empty:
                     writer.maybe_flush()
                     continue
+                kind = item[0]
+                seq = item[1]
+                self._flush_gaps(writer, seq)
                 if kind == "stop":
                     break
                 if kind == "event":
-                    self._handle_event(writer, payload)
+                    self._handle_event(writer, item[2])
                     continue
-                sample, frame = payload
+                _kind, _seq, sample, frame = item
                 self._handle_frame(writer, sample, frame)
                 writer.maybe_flush()
         except Exception as exc:  # noqa: BLE001 - reported through stats
@@ -617,32 +713,31 @@ class ExperimentRecorder:
                 self._accepting = False
             problems: list[str] = []
             if writer is not None:
-                # A packet accepted just before the stop flag may have left its
-                # row or gap behind; write those before the file is closed.
-                self._drain_rows(writer)
-                self._drain_gaps(writer)
+                # A row or gap accepted just before the stop flag may still be in
+                # the stream; write those before the file is closed.
+                self._flush_gaps(writer, None)
                 try:
                     problems = writer.close()
                 except Exception as exc:  # noqa: BLE001
                     problems = [f"close failed: {type(exc).__name__}: {exc}"]
                 with self._lock:
-                    self._stats.write_errors += writer.write_errors
                     self._stats.segments = writer.segments
                     self._stats.video_frames = writer.video_frames
                     self._stats.bytes_written = writer.bytes_written
+            # Closing problems are failures of the same recording and are counted
+            # exactly once, through the same path as a failed row, flush or event.
+            for problem in problems:
+                self._record_failure(problem)
+                self._alarm(problem)
             with self._lock:
-                if problems or failure:
-                    self._stats.failed = True
-                    if not self._stats.last_error:
-                        self._stats.last_error = problems[0] if problems else failure
+                if failure and not self._stats.last_error:
+                    self._stats.last_error = failure
                 # The writer's own findings (a flush that failed, a pre-roll that
                 # could not be stored) belong in the report: an operator reading
                 # only the terminal report must not have to grep the log.
                 self._writer_problems = list(problems)
                 failed = self._stats.failed
                 self._state = STATE_FAILED if failed else STATE_STOPPED
-            for problem in problems:
-                self._alarm(problem)
             # Only now, with the files closed, is the report trustworthy.
             self._report = self._finalise(completed=True)
 
@@ -657,6 +752,10 @@ class ExperimentRecorder:
     def _handle_frame(self, writer, sample, frame) -> None:
         try:
             if frame is not None:
+                # The one release for the reservation taken in ``on_packet``.
+                # A frame is enqueued once and consumed once, so this cannot
+                # double-count; doing it *before* the write also means a failed
+                # write does not leak the budget for a frame that is gone.
                 with self._lock:
                     self._queued_bytes = max(0, self._queued_bytes - int(frame.nbytes))
             writer.write_sample(sample, frame)
@@ -668,22 +767,21 @@ class ExperimentRecorder:
         except Exception as exc:  # noqa: BLE001
             self._record_error(exc)
 
-    def _drain_rows(self, writer) -> None:
-        """Write rows parked while the item queue was full of video."""
-        while True:
-            with self._lock:
-                if not self._row_backlog:
-                    return
-                sample = self._row_backlog.popleft()
-            self._handle_frame(writer, sample, None)
+    def _flush_gaps(self, writer, upto_seq) -> None:
+        """Write the frame-index rows for frames whose data row was lost.
 
-    def _drain_gaps(self, writer) -> None:
-        """Write the frame-index rows for frames whose data row was lost."""
+        Gaps are written in stream order and only up to ``upto_seq`` (``None``
+        flushes the rest), so ``frames.csv`` stays sorted by time even when the
+        loss happened while the writer was far behind.
+        """
         while True:
             with self._lock:
                 if not self._gaps:
                     return
-                frame_id, mono_t, wall_t = self._gaps.popleft()
+                seq, frame_id, mono_t, wall_t = self._gaps[0]
+                if upto_seq is not None and seq > upto_seq:
+                    return
+                self._gaps.popleft()
             try:
                 writer.write_gap(frame_id, mono_t, wall_t)
                 with self._lock:
@@ -693,18 +791,32 @@ class ExperimentRecorder:
                 return
 
     def _record_error(self, exc: Exception) -> None:
+        self._record_failure(f"{type(exc).__name__}: {exc}")
+
+    def _record_failure(self, message: str, *, fatal: bool = False) -> None:
+        """One failed write, flush, close or metadata update.
+
+        Every failure enters the state machine here, so the counters, the fatal
+        transition and the report cannot disagree about what happened. A writer
+        that kept its own count (a failed ``maybe_flush`` used to) left the
+        recording looking healthy until the very end.
+
+        ``fatal`` marks a failure that ends the recording on its own: a recording
+        whose ``session.json`` could not be written is not a recording anyone can
+        identify afterwards.
+        """
         with self._lock:
             self._stats.write_errors += 1
-            self._stats.last_error = f"{type(exc).__name__}: {exc}"
+            self._stats.last_error = str(message)
             self._consecutive_errors += 1
-            fatal = self._consecutive_errors >= MAX_CONSECUTIVE_ERRORS
+            fatal = fatal or self._consecutive_errors >= MAX_CONSECUTIVE_ERRORS
             if fatal:
                 self._stats.failed = True
                 # Stop accepting immediately: consuming packets that cannot be
                 # written only hides the loss behind a "recording" state.
                 self._accepting = False
                 self._state = STATE_FAILED
-        self._alarm(f"recording write failed: {exc}")
+        self._alarm(f"recording write failed: {message}")
         if fatal:
             self._stop.set()
 
@@ -712,6 +824,10 @@ class ExperimentRecorder:
         with self._lock:
             if self._t0 is not None and self._last_t is not None:
                 self._stats.duration_s = round(float(self._last_t - self._t0), 6)
+        # Metadata **before** the report is assembled: a session.json that could
+        # not be written is part of what happened, and used to be discovered only
+        # after the report had already been built from the older statistics.
+        self._write_session_json(final=completed)
         stats = self.stats()
         problems: list[str] = []
         if stats.dropped_data or stats.dropped_video:
@@ -734,8 +850,6 @@ class ExperimentRecorder:
             for problem in self._writer_problems:
                 if problem not in problems:
                     problems.append(problem)
-        # ``ended_at`` is only written once the files are actually closed.
-        self._write_session_json(final=completed)
         return RecordReport(
             directory=self._directory,
             stats=stats,
@@ -763,10 +877,11 @@ class ExperimentRecorder:
                 )
                 temporary.replace(target)
         except Exception as exc:  # noqa: BLE001
-            with self._lock:
-                self._stats.write_errors += 1
-                self._stats.last_error = f"session.json: {type(exc).__name__}: {exc}"
-            self._alarm(f"could not write session.json: {exc}")
+            # The same single failure path as a write: it is counted, it ends the
+            # recording, and it reaches the report.
+            self._record_failure(
+                f"session.json: {type(exc).__name__}: {exc}", fatal=True
+            )
 
     def _build_metadata(self, *, final: bool) -> dict[str, Any]:
         with self._lock:
@@ -783,6 +898,9 @@ class ExperimentRecorder:
             "video": self.config.as_dict(),
             "events": events,
             "stats": stats,
+            # The coefficients the session actually ran with: the file itself is
+            # copied into the recording, so a re-export cannot rewrite history.
+            "calibration_snapshots": [dict(entry) for entry in self._calibration_snapshots],
             # The replay needs these to know whether the algorithm state at the
             # start of the recording can be rebuilt at all.
             "preroll_frames": int(self._preroll_frames),
@@ -802,16 +920,36 @@ class ExperimentRecorder:
 class _SessionWriter:
     """Owns the file handles. Only ever touched by the writer thread."""
 
+    def _note_problem(self, message: str) -> None:
+        """Record a failure here *and* hand it to the recorder at once."""
+        self.write_errors += 1
+        if len(self._problems) < 5:
+            self._problems.append(message)
+        if self._on_problem is not None:
+            try:
+                self._on_problem(message)
+            except Exception:  # noqa: BLE001 - reporting must not break writing
+                logger.exception("Recording problem handler failed")
+
     def __init__(
-        self, *, directory, config, session_id, calibration_id, preroll=()
+        self,
+        *,
+        directory,
+        config,
+        session_id,
+        calibration_id,
+        preroll=(),
+        on_problem=None,
     ) -> None:
         self.directory = Path(directory)
         self.config = config
         self.session_id = session_id
         self.calibration_id = calibration_id
+        #: ``on_problem(message)`` reports a failure to the recorder's state
+        #: machine *immediately*. A private counter here was invisible until the
+        #: files were closed, so a failing flush looked like a healthy recording.
+        self._on_problem = on_problem
         self.write_errors = 0
-        self.segments = 0
-        self.video_frames = 0
         self.png_frames = 0
         self.gap_rows = 0
         self.bytes_written = 0
@@ -820,12 +958,14 @@ class _SessionWriter:
         self.preroll_segment: Optional[str] = None
         self._last_flush = time.perf_counter()
         self._t0: Optional[float] = None
-        self._segment: Optional[cv2.VideoWriter] = None
-        self._segment_index = 0
-        self._segment_frames = 0
-        self._segment_started = 0.0
-        self._frame_shape: Optional[tuple[int, int]] = None
         self._problems: list[str] = []
+        #: The video side, in its own component: one rolling segment at a time.
+        self._video = VideoSegments(
+            self.directory,
+            codec=self.config.codec,
+            fps=self.config.fps,
+            segment_seconds=self.config.segment_seconds,
+        )
 
         self._forces = self._open_csv("forces.csv", FORCES_HEADER)
         self._frames = self._open_csv("frames.csv", FRAMES_HEADER)
@@ -863,22 +1003,8 @@ class _SessionWriter:
             kept.append(frame)
             used += size
         kept.reverse()
-        self._frame_shape = (int(kept[0].shape[1]), int(kept[0].shape[0]))
-        name = "preroll_0001.mp4"
-        path = self.directory / name
-        fourcc = cv2.VideoWriter_fourcc(*self.config.codec)
-        writer = cv2.VideoWriter(
-            str(path), fourcc, max(1.0, float(self.config.fps)), self._frame_shape
-        )
-        if not writer.isOpened():
-            raise OSError(f"could not open video writer for {name}")
-        try:
-            for frame in kept:
-                writer.write(frame)
-                self.preroll_frames += 1
-        finally:
-            writer.release()
-        self.preroll_segment = name if self.preroll_frames else None
+        self.preroll_frames = self._video.write_sequence(kept)
+        self.preroll_segment = "preroll_0001.mp4" if self.preroll_frames else None
 
     def _open_csv(self, name: str, header):
         handle = (self.directory / name).open("w", newline="", encoding="utf-8")
@@ -957,16 +1083,15 @@ class _SessionWriter:
         self.gap_rows += 1
 
     def _write_video_frame(self, frame) -> tuple[int, int]:
-        if self._frame_shape is None:
-            self._frame_shape = (int(frame.shape[1]), int(frame.shape[0]))
-        if self._should_roll():
-            self._close_segment()
-        if self._segment is None:
-            self._open_segment()
-        self._segment.write(frame)
-        self._segment_frames += 1
-        self.video_frames += 1
-        return (self._segment_index, self._segment_frames)
+        return self._video.write(frame)
+
+    @property
+    def video_frames(self) -> int:
+        return self._video.video_frames
+
+    @property
+    def segments(self) -> int:
+        return self._video.segments
 
     def _write_lossless(self, frame) -> None:
         self.png_frames += 1
@@ -989,33 +1114,6 @@ class _SessionWriter:
             )
         )
 
-    # ------------------------------------------------------------------ segments
-    def _should_roll(self) -> bool:
-        if self._segment is None:
-            return False
-        if self.config.segment_seconds <= 0:
-            return False
-        return (time.perf_counter() - self._segment_started) >= self.config.segment_seconds
-
-    def _open_segment(self) -> None:
-        self._segment_index += 1
-        path = self.directory / f"raw_{self._segment_index:04d}.mp4"
-        fourcc = cv2.VideoWriter_fourcc(*self.config.codec)
-        fps = max(1.0, float(self.config.fps))
-        writer = cv2.VideoWriter(str(path), fourcc, fps, self._frame_shape)
-        if not writer.isOpened():
-            raise OSError(f"could not open video writer for {path.name}")
-        self._segment = writer
-        self._segment_frames = 0
-        self._segment_started = time.perf_counter()
-        self.segments = self._segment_index
-
-    def _close_segment(self) -> None:
-        if self._segment is None:
-            return
-        self._segment.release()
-        self._segment = None
-
     def maybe_flush(self) -> None:
         if (time.perf_counter() - self._last_flush) < self.config.flush_interval_s:
             return
@@ -1024,23 +1122,22 @@ class _SessionWriter:
             try:
                 handle.flush()
             except Exception as exc:  # noqa: BLE001
-                self.write_errors += 1
-                if len(self._problems) < 5:
-                    self._problems.append(f"flush failed: {exc}")
+                self._note_problem(f"flush failed: {exc}")
 
     def close(self) -> list[str]:
-        self._close_segment()
+        try:
+            self._video.close()
+        except Exception as exc:  # noqa: BLE001 - the CSVs still have to close
+            self._note_problem(f"video close failed: {exc}")
         for handle, _writer in (self._forces, self._frames, self._events):
             try:
                 handle.flush()
                 handle.close()
             except Exception as exc:  # noqa: BLE001
-                self.write_errors += 1
-                if len(self._problems) < 5:
-                    self._problems.append(f"close failed: {exc}")
+                self._note_problem(f"close failed: {exc}")
         try:
             self.bytes_written = sum(
-                path.stat().st_size for path in self.directory.glob("*") if path.is_file()
+                path.stat().st_size for path in self.directory.rglob("*") if path.is_file()
             )
         except Exception:  # noqa: BLE001
             self.bytes_written = 0

shared/src/orisys_app_shared/record/index.py

@@ -0,0 +1,453 @@
+"""A time index over a recorded ``forces.csv``, so history is read on demand.
+
+The replay used to load every row of a recording into Python lists and then into
+arrays. That is fine for a minute and hopeless for the multi-hour recordings the
+store is designed for, and it made the charts the only place where "the whole
+session" had to exist in memory at once.
+
+This index keeps the *small* things in memory — one time per row, one validity
+flag, one frame id, plus periodic byte offsets — and reads the *values* only for
+the interval being drawn. It also builds a multi-level overview (per-bucket
+min/max/count) while streaming the file once, which is what a whole-session strip
+needs and what it cannot get by sampling a few thousand rows.
+
+Nothing here touches Qt, and nothing here trusts the file to be in order: a
+legacy recording is sorted at load time (see ``playback.RecordingReader``).
+"""
+
+from __future__ import annotations
+
+import csv
+import logging
+from bisect import bisect_left, bisect_right
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Iterable, Iterator, Optional
+
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+#: How many rows to read between two recorded byte offsets. Larger means a
+#: cheaper interval read but a longer scan to reach the interval.
+CHECKPOINT_ROWS = 4096
+#: Default number of overview buckets for the whole-session strip.
+DEFAULT_BUCKETS = 1200
+
+_FORCE_COLUMNS = ("FN_raw", "FX_raw", "FY_raw", "FN_N", "FX_N", "FY_N")
+
+
+def _parse(text: str) -> float:
+    text = (text or "").strip()
+    if not text:
+        # An empty cell means "not available"; it must never become 0.
+        return float("nan")
+    try:
+        return float(text)
+    except ValueError:
+        return float("nan")
+
+
+@dataclass
+class ForceWindow:
+    """Rows inside one interval — the same shape a session slice has."""
+
+    times: np.ndarray
+    raw: np.ndarray
+    newtons: np.ndarray
+    valid: np.ndarray
+    frame_ids: np.ndarray
+    #: True when the rows were decimated to satisfy the caller's point budget.
+    decimated: bool = False
+
+    def __len__(self) -> int:
+        return int(self.times.size)
+
+    def value_for_unit(self, unit: str) -> np.ndarray:
+        if str(unit).upper().startswith("N"):
+            return self.newtons
+        return self.raw
+
+    def plottable(self, unit: str):
+        """``(times, values, mask)`` with a per-axis mask — the shared rule."""
+        from orisys_app_shared.record.sample import plottable_axis_mask
+
+        values = self.value_for_unit(unit)
+        return self.times, values, plottable_axis_mask(values, self.valid)
+
+    def plottable_axis(self, unit: str, index: int):
+        from orisys_app_shared.record.sample import plottable_axis_mask
+
+        values = self.value_for_unit(unit)
+        mask = plottable_axis_mask(values, self.valid)[:, index]
+        return self.times[mask], values[mask, index]
+
+    def statistics(self, unit: str) -> "WindowStats":
+        values = self.value_for_unit(unit)
+        mask = np.isfinite(values) & self.valid[:, None]
+        peaks: list[float] = []
+        means: list[float] = []
+        stds: list[float] = []
+        for index in range(3):
+            column = values[:, index][mask[:, index]]
+            if column.size == 0:
+                peaks.append(float("nan"))
+                means.append(float("nan"))
+                stds.append(float("nan"))
+                continue
+            peaks.append(float(np.nanmax(column)))
+            means.append(float(np.nanmean(column)))
+            stds.append(float(np.nanstd(column)))
+        span = 0.0
+        if self.times.size > 1:
+            span = float(self.times[-1] - self.times[0])
+        return WindowStats(
+            duration_s=span,
+            samples=int(mask.any(axis=1).sum()),
+            peaks=tuple(peaks),
+            means=tuple(means),
+            stds=tuple(stds),
+        )
+
+
+@dataclass(frozen=True)
+class WindowStats:
+    duration_s: float = 0.0
+    samples: int = 0
+    peaks: tuple[float, float, float] = (float("nan"),) * 3
+    means: tuple[float, float, float] = (float("nan"),) * 3
+    stds: tuple[float, float, float] = (float("nan"),) * 3
+
+
+@dataclass
+class Overview:
+    """A whole-session envelope, cheap enough to keep in memory."""
+
+    times: np.ndarray
+    lows: np.ndarray  # (buckets, 3)
+    highs: np.ndarray  # (buckets, 3)
+    counts: np.ndarray  # (buckets,)
+    unit: str = "raw"
+
+    def __len__(self) -> int:
+        return int(self.times.size)
+
+
+@dataclass
+class _Checkpoint:
+    row: int
+    time: float
+    offset: int
+
+
+class RecordingIndex:
+    """Index one recording directory's ``forces.csv`` (and its validity flags)."""
+
+    def __init__(self, directory: str | Path, *, checkpoint_rows: int = CHECKPOINT_ROWS):
+        self.directory = Path(directory)
+        self.checkpoint_rows = max(1, int(checkpoint_rows))
+        self.times = np.empty(0, dtype=np.float64)
+        self.valid = np.empty(0, dtype=bool)
+        self.frame_ids = np.empty(0, dtype=np.int64)
+        self._checkpoints: list[_Checkpoint] = []
+        self._rows = 0
+        self._byte_size = 0
+        self._built = False
+
+    # ------------------------------------------------------------------ building
+    @property
+    def path(self) -> Path:
+        return self.directory / "forces.csv"
+
+    @property
+    def rows(self) -> int:
+        return self._rows
+
+    @property
+    def byte_size(self) -> int:
+        """Size of the data file the index describes."""
+        return self._byte_size
+
+    def build(self) -> "RecordingIndex":
+        """One streaming pass: times, flags, frame ids and byte offsets."""
+        path = self.path
+        times: list[float] = []
+        valid: list[bool] = []
+        frame_ids: list[int] = []
+        checkpoints: list[_Checkpoint] = []
+        if path.is_file():
+            with path.open("rb") as handle:
+                header = handle.readline().decode("utf-8", "replace").strip()
+                columns = {name: index for index, name in enumerate(header.split(","))}
+                time_column = columns.get("t_rel_s", 2)
+                valid_column = columns.get("valid", 9)
+                frame_column = columns.get("frame_id", 1)
+                offset = handle.tell()
+                for row, line in enumerate(handle):
+                    next_offset = handle.tell()
+                    text = line.decode("utf-8", "replace").rstrip("\r\n")
+                    if row % self.checkpoint_rows == 0:
+                        checkpoints.append(
+                            _Checkpoint(row=row, time=_time_of(text), offset=offset)
+                        )
+                    fields = next(csv.reader([text])) if text else []
+                    if len(fields) > max(time_column, valid_column, frame_column):
+                        try:
+                            times.append(float(fields[time_column]))
+                            valid.append(fields[valid_column].strip() == "1")
+                            frame_ids.append(int(fields[frame_column]))
+                        except (TypeError, ValueError):
+                            times.append(float("nan"))
+                            valid.append(False)
+                            frame_ids.append(-1)
+                    offset = next_offset
+        self.times = np.asarray(times, dtype=np.float64)
+        self.valid = np.asarray(valid, dtype=bool)
+        self.frame_ids = np.asarray(frame_ids, dtype=np.int64)
+        self._checkpoints = checkpoints
+        self._rows = int(self.times.size)
+        try:
+            self._byte_size = path.stat().st_size
+        except OSError:
+            self._byte_size = 0
+        self._built = True
+        return self
+
+    def ensure_built(self) -> "RecordingIndex":
+        if not self._built:
+            self.build()
+        return self
+
+    # ------------------------------------------------------------------- queries
+    def extent(self) -> tuple[float, float]:
+        if self._rows == 0:
+            return (0.0, 0.0)
+        return (float(self.times[0]), float(self.times[-1]))
+
+    def duration_s(self) -> float:
+        first, last = self.extent()
+        return max(0.0, last - first)
+
+    def row_range(self, t0: float, t1: float) -> tuple[int, int]:
+        """Half-open row range covering ``[t0, t1]``."""
+        self.ensure_built()
+        if self._rows == 0:
+            return (0, 0)
+        if float(t1) < float(t0):
+            t0, t1 = t1, t0
+        start = max(0, bisect_left(self.times.tolist(), float(t0)) - 1)
+        stop = min(self._rows, bisect_right(self.times.tolist(), float(t1)))
+        return (start, max(start, stop))
+
+    def _checkpoint_before(self, row: int) -> _Checkpoint:
+        chosen = _Checkpoint(row=0, time=0.0, offset=0)
+        for checkpoint in self._checkpoints:
+            if checkpoint.row <= row:
+                chosen = checkpoint
+            else:
+                break
+        return chosen
+
+    def read_rows(
+        self,
+        start: int,
+        stop: int,
+        *,
+        max_points: int | None = None,
+        unit: str = "raw",
+    ) -> ForceWindow:
+        """Read ``[start, stop)`` by seeking to the nearest checkpoint.
+
+        With ``max_points`` the rows are decimated **while streaming**: each
+        bucket contributes the extreme row of every axis in ``unit``, so a long
+        interval never has to exist in memory in full. The extremes are taken in
+        both the raw and the requested unit's axis, which is why a shear spike
+        survives a flat normal channel.
+        """
+        self.ensure_built()
+        start = max(0, min(int(start), self._rows))
+        stop = max(start, min(int(stop), self._rows))
+        if stop <= start:
+            return _empty_window()
+        path = self.path
+        if not path.is_file():
+            return _empty_window()
+
+        newton_unit = str(unit).upper().startswith("N")
+        budget = int(max_points) if max_points else 0
+        total = stop - start
+        bucket = (
+            max(1, total // max(1, budget // 6))
+            if budget and total > budget * 2
+            else 1
+        )
+
+        checkpoint = self._checkpoint_before(start)
+        pending: list[tuple] = []
+        picked: list[tuple] = []
+
+        def flush() -> None:
+            if not pending:
+                return
+            if bucket == 1:
+                picked.extend(pending)
+            else:
+                values = np.asarray(
+                    [row[3] if newton_unit else row[2] for row in pending],
+                    dtype=np.float64,
+                )
+                for axis in range(3):
+                    column = values[:, axis]
+                    finite = np.isfinite(column)
+                    if not finite.any():
+                        continue
+                    ordered = np.where(finite, column, -np.inf)
+                    for index in {int(np.argmin(ordered)), int(np.argmax(ordered))}:
+                        picked.append(pending[index])
+            pending.clear()
+
+        with path.open("rb") as handle:
+            # Every checkpoint offset points at a *data* row: the build records
+            # ``tell()`` after the header, so there is nothing to skip here.
+            handle.seek(checkpoint.offset)
+            for row, line in enumerate(handle, start=checkpoint.row):
+                if row >= stop:
+                    break
+                if row < start:
+                    continue
+                fields = next(csv.reader([line.decode("utf-8", "replace")]))
+                if len(fields) < 11:
+                    continue
+                try:
+                    t_rel = float(fields[2])
+                    frame_id = int(fields[1])
+                except (TypeError, ValueError):
+                    continue
+                values = [_parse(fields[index]) for index in (3, 4, 5, 6, 7, 8)]
+                pending.append(
+                    (
+                        frame_id,
+                        t_rel,
+                        tuple(values[:3]),
+                        tuple(values[3:]),
+                        fields[9].strip() == "1",
+                    )
+                )
+                if len(pending) >= bucket:
+                    flush()
+        flush()
+
+        if not picked:
+            return _empty_window()
+        # A row can be the extreme of several axes; keep one copy, in time order.
+        unique = {row[0]: row for row in picked}
+        rows = sorted(unique.values(), key=lambda row: row[1])
+        return ForceWindow(
+            times=np.asarray([row[1] for row in rows], dtype=np.float64),
+            raw=np.asarray([row[2] for row in rows], dtype=np.float64).reshape(-1, 3),
+            newtons=np.asarray([row[3] for row in rows], dtype=np.float64).reshape(-1, 3),
+            valid=np.asarray([row[4] for row in rows], dtype=bool),
+            frame_ids=np.asarray([row[0] for row in rows], dtype=np.int64),
+            decimated=bucket > 1,
+        )
+
+    def iter_rows(self, start: int = 0, stop: Optional[int] = None) -> Iterator[tuple]:
+        """Yield ``(row, frame_id, t_rel_s, raw3, newtons3, valid)`` in file order."""
+        self.ensure_built()
+        path = self.path
+        if not path.is_file():
+            return
+        stop = self._rows if stop is None else min(int(stop), self._rows)
+        with path.open("rb") as handle:
+            handle.readline()  # header
+            for row, line in enumerate(handle):
+                if row >= stop:
+                    return
+                if row < start:
+                    continue
+                fields = next(csv.reader([line.decode("utf-8", "replace")]))
+                if len(fields) < 11:
+                    continue
+                try:
+                    frame_id = int(fields[1])
+                    t_rel = float(fields[2])
+                except (TypeError, ValueError):
+                    continue
+                values = [_parse(fields[index]) for index in (3, 4, 5, 6, 7, 8)]
+                yield (
+                    row,
+                    frame_id,
+                    t_rel,
+                    tuple(values[:3]),
+                    tuple(values[3:]),
+                    fields[9].strip() == "1",
+                )
+
+    # ------------------------------------------------------------------ overview
+    def overview(self, *, unit: str = "raw", buckets: int = DEFAULT_BUCKETS) -> Overview:
+        """A per-bucket envelope of the whole recording, in the active unit.
+
+        Built by streaming the file once and folding each row into its bucket, so
+        a peak in a bucket the strip would otherwise skip is still visible.
+        """
+        self.ensure_built()
+        buckets = max(1, int(buckets))
+        newton = str(unit).upper().startswith("N")
+        if self._rows == 0:
+            return Overview(
+                times=np.empty(0),
+                lows=np.empty((0, 3)),
+                highs=np.empty((0, 3)),
+                counts=np.empty(0, dtype=np.int64),
+                unit="N" if newton else "raw",
+            )
+        first, last = self.extent()
+        span = max(1e-9, last - first)
+        lows = np.full((buckets, 3), np.nan)
+        highs = np.full((buckets, 3), np.nan)
+        counts = np.zeros(buckets, dtype=np.int64)
+        starts = np.full(buckets, np.nan)
+        for _row, _frame_id, t_rel, raw, newtons, is_valid in self.iter_rows():
+            if not is_valid:
+                continue
+            bucket = int((t_rel - first) / span * (buckets - 1) + 0.5)
+            bucket = max(0, min(buckets - 1, bucket))
+            values = newtons if newton else raw
+            for axis in range(3):
+                value = values[axis]
+                if value != value:  # NaN
+                    continue
+                current_low = lows[bucket, axis]
+                current_high = highs[bucket, axis]
+                lows[bucket, axis] = value if current_low != current_low else min(current_low, value)
+                highs[bucket, axis] = value if current_high != current_high else max(current_high, value)
+            if starts[bucket] != starts[bucket]:
+                starts[bucket] = t_rel
+            counts[bucket] += 1
+        return Overview(
+            times=starts,
+            lows=lows,
+            highs=highs,
+            counts=counts,
+            unit="N" if newton else "raw",
+        )
+
+
+def _time_of(text: str) -> float:
+    fields = next(csv.reader([text])) if text else []
+    if len(fields) > 2:
+        try:
+            return float(fields[2])
+        except (TypeError, ValueError):
+            return float("nan")
+    return float("nan")
+
+
+def _empty_window() -> ForceWindow:
+    return ForceWindow(
+        times=np.empty(0, dtype=np.float64),
+        raw=np.empty((0, 3)),
+        newtons=np.empty((0, 3)),
+        valid=np.empty(0, dtype=bool),
+        frame_ids=np.empty(0, dtype=np.int64),
+    )

shared/src/orisys_app_shared/record/playback.py

@@ -21,6 +21,15 @@ from typing import Iterator, Optional
 
 import numpy as np
 
+from orisys_app_shared.record.index import (
+    CHECKPOINT_ROWS,
+    DEFAULT_BUCKETS,
+    ForceWindow,
+    Overview,
+    RecordingIndex,
+)
+from orisys_app_shared.record.sample import plottable_axis_mask
+
 logger = logging.getLogger(__name__)
 
 #: Column names as written by the recorder (index == position in forces.csv).
@@ -65,9 +74,21 @@ class RecordingInfo:
     frame_ids: np.ndarray
     dropped_rows: int = 0
     frames_dropped: int = 0
+    #: The index rows were in time order as written (False = a legacy record).
+    frames_out_of_order: bool = True
+    #: The force rows were in time order as written.
+    forces_out_of_order: bool = True
+    #: True when the arrays above are a decimated view of a long recording.
+    decimated: bool = False
+    #: Total rows in ``forces.csv`` (the decimated view has fewer).
+    total_rows: int = 0
+    #: The recording's real duration, from the frame index, never from the view.
+    total_duration_s: float = 0.0
 
     @property
     def duration_s(self) -> float:
+        if self.total_duration_s:
+            return float(self.total_duration_s)
         if self.times.size == 0:
             return 0.0
         return float(self.times[-1])
@@ -78,26 +99,51 @@ class RecordingInfo:
         return self.raw
 
     def plottable(self, unit: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
-        """``(times, values, mask)`` — the same validity rule the live charts use.
+        """``(times, values, mask)`` — the same rule the live charts use.
 
         The replay used to filter on finiteness alone, so a stale value left by a
         failed compute step could reappear here after another view had already
-        excluded it.
+        excluded it. ``mask`` is per axis: a single-axis calibration is normal and
+        must not hide the axis that *is* calibrated.
         """
         values = self.value_for_unit(unit)
-        mask = self.valid & np.isfinite(values).all(axis=1)
-        return self.times[mask], values[mask], mask
+        return self.times, values, plottable_axis_mask(values, self.valid)
+
+    def plottable_axis(self, unit: str, index: int) -> tuple[np.ndarray, np.ndarray]:
+        """``(times, values)`` for one channel, with the other axes ignored."""
+        values = self.value_for_unit(unit)
+        mask = plottable_axis_mask(values, self.valid)[:, index]
+        return self.times[mask], values[mask, index]
 
 
 class RecordingReader:
-    """Open one experiment directory and serve replay queries from it."""
+    """Open one experiment directory and serve replay queries from it.
 
-    def __init__(self, directory: str | Path) -> None:
+    ``max_rows`` bounds what :meth:`info` keeps in memory: a long recording is
+    loaded as a decimated view (peaks preserved) and marked ``decimated``, while
+    :meth:`window` and :meth:`overview` read the interval being drawn from disk
+    through :class:`~orisys_app_shared.record.index.RecordingIndex`. That is what
+    lets the replay show an hour-long recording without holding all of it.
+    """
+
+    #: Above this many rows ``info()`` decimates instead of loading everything.
+    DEFAULT_MAX_ROWS = 200_000
+
+    def __init__(
+        self,
+        directory: str | Path,
+        *,
+        max_rows: int | None = None,
+        checkpoint_rows: int = CHECKPOINT_ROWS,
+    ) -> None:
         self.directory = Path(directory)
         if not self.directory.is_dir():
             raise FileNotFoundError(f"Not a recording directory: {self.directory}")
         self._info: Optional[RecordingInfo] = None
         self._events: Optional[list[dict]] = None
+        self.max_rows = int(max_rows or self.DEFAULT_MAX_ROWS)
+        self._index = RecordingIndex(self.directory, checkpoint_rows=checkpoint_rows)
+        self._overviews: dict[tuple[str, int], Overview] = {}
 
     # ------------------------------------------------------------------ loading
     def info(self) -> RecordingInfo:
@@ -105,6 +151,10 @@ class RecordingReader:
             self._info = self._load()
         return self._info
 
+    def index(self) -> RecordingIndex:
+        """The time index, built once and shared with the windows."""
+        return self._index.ensure_built()
+
     def session(self) -> dict:
         path = self.directory / "session.json"
         if not path.is_file():
@@ -116,8 +166,30 @@ class RecordingReader:
             return {}
 
     def _load(self) -> RecordingInfo:
-        frames = self._load_frames()
-        times, raw, newtons, valid, frame_ids, dropped = self._load_forces()
+        frames, frames_ordered = self._load_frames()
+        index = self.index()
+        total_rows = index.rows
+        decimated = total_rows > self.max_rows > 0
+        if decimated:
+            window = self.window_all(
+                max_points=self.max_rows
+            )
+            times = window.times
+            raw = window.raw
+            newtons = window.newtons
+            valid = window.valid
+            frame_ids = window.frame_ids
+            forces_ordered = True
+            dropped = 0
+        else:
+            (times, raw, newtons, valid, frame_ids, dropped, forces_ordered) = (
+                self._load_forces()
+            )
+        total_duration = 0.0
+        if frames:
+            total_duration = float(frames[-1].t_rel_s)
+        elif times.size:
+            total_duration = float(times[-1])
         return RecordingInfo(
             directory=self.directory,
             session=self.session(),
@@ -129,12 +201,26 @@ class RecordingReader:
             frame_ids=frame_ids,
             dropped_rows=dropped,
             frames_dropped=sum(1 for frame in frames if frame.dropped),
+            frames_out_of_order=frames_ordered,
+            forces_out_of_order=forces_ordered,
+            decimated=decimated,
+            total_rows=total_rows,
+            total_duration_s=total_duration,
         )
 
-    def _load_frames(self) -> list[RecordingFrame]:
+    def _load_frames(self) -> tuple[list[RecordingFrame], bool]:
+        """Load the frame index in time order.
+
+        Recordings made before the ordered-write fix (see
+        ``ExperimentRecorder``) can contain a frame index whose rows are not in
+        time order, because the row backlog used to be drained before the queue.
+        ``frame_at`` does a binary search over these times, so an unsorted index
+        would silently answer with the wrong frame. The rows are sorted here and
+        the fact is reported as a replay warning.
+        """
         path = self.directory / "frames.csv"
         if not path.is_file():
-            return []
+            return [], False
         rows: list[RecordingFrame] = []
         with path.open(encoding="utf-8", newline="") as handle:
             for row in csv.DictReader(handle):
@@ -152,13 +238,26 @@ class RecordingReader:
                     )
                 except (KeyError, TypeError, ValueError):
                     continue
-        return rows
+        ordered = _is_time_ordered([frame.t_rel_s for frame in rows])
+        if not ordered:
+            rows.sort(key=lambda frame: frame.t_rel_s)
+        return rows, ordered
 
     def _load_forces(self):
         path = self.directory / "forces.csv"
         if not path.is_file():
+            # A directory with no data (an empty or half-written recording) is
+            # readable: it simply has no samples, and nothing is out of order.
             empty = np.empty(0, dtype=np.float64)
-            return empty, np.empty((0, 3)), np.empty((0, 3)), np.empty(0, bool), np.empty(0, int), 0
+            return (
+                empty,
+                np.empty((0, 3)),
+                np.empty((0, 3)),
+                np.empty(0, bool),
+                np.empty(0, int),
+                0,
+                True,
+            )
 
         def parse(text: str) -> float:
             text = (text or "").strip()
@@ -191,23 +290,42 @@ class RecordingReader:
                     (parse(row.get("FN_N")), parse(row.get("FX_N")), parse(row.get("FY_N")))
                 )
                 valid.append((row.get("valid") or "0").strip() == "1")
-        return (
-            np.asarray(times, dtype=np.float64),
-            np.asarray(raw, dtype=np.float64).reshape(-1, 3),
-            np.asarray(newtons, dtype=np.float64).reshape(-1, 3),
-            np.asarray(valid, dtype=bool),
-            np.asarray(frame_ids, dtype=np.int64),
-            dropped,
-        )
+        times_arr = np.asarray(times, dtype=np.float64)
+        raw_arr = np.asarray(raw, dtype=np.float64).reshape(-1, 3)
+        newtons_arr = np.asarray(newtons, dtype=np.float64).reshape(-1, 3)
+        valid_arr = np.asarray(valid, dtype=bool)
+        ids_arr = np.asarray(frame_ids, dtype=np.int64)
+
+        # Older recordings can have rows out of time order (the writer used to
+        # drain a backlog before the queue). Every window query below assumes
+        # sorted time, so sort once here and report it instead of answering
+        # ``searchsorted`` on unsorted input.
+        ordered = _is_time_ordered(times)
+        if not ordered and times_arr.size:
+            order = np.argsort(times_arr, kind="stable")
+            times_arr = times_arr[order]
+            raw_arr = raw_arr[order]
+            newtons_arr = newtons_arr[order]
+            valid_arr = valid_arr[order]
+            ids_arr = ids_arr[order]
+        return times_arr, raw_arr, newtons_arr, valid_arr, ids_arr, dropped, ordered
 
     # ------------------------------------------------------------------ queries
     def frame_at(self, t_rel_s: float) -> Optional[RecordingFrame]:
-        """The recorded frame at or just before ``t_rel_s``."""
+        """The recorded frame at or just before ``t_rel_s``.
+
+        The index is sorted at load time, so the binary search is valid; the
+        fallback walk keeps a hand-edited or truncated file from answering with
+        the wrong frame.
+        """
         frames = self.info().frames
         if not frames:
             return None
-        times = [frame.t_rel_s for frame in frames]
-        index = int(np.searchsorted(np.asarray(times), float(t_rel_s), side="right")) - 1
+        times = np.asarray([frame.t_rel_s for frame in frames], dtype=np.float64)
+        if not _is_time_ordered(times.tolist()):
+            candidates = [frame for frame in frames if frame.t_rel_s <= float(t_rel_s)]
+            return max(candidates, key=lambda frame: frame.t_rel_s) if candidates else frames[0]
+        index = int(np.searchsorted(times, float(t_rel_s), side="right")) - 1
         index = max(0, min(index, len(frames) - 1))
         return frames[index]
 
@@ -318,6 +436,47 @@ class RecordingReader:
         mask = (info.times >= float(t0)) & (info.times <= float(t1))
         return info.times[mask], info.raw[mask]
 
+    # ------------------------------------------------------ on-demand history
+    def window(
+        self,
+        t0: float,
+        t1: float,
+        *,
+        unit: str = "raw",
+        max_points: int | None = None,
+    ) -> ForceWindow:
+        """The recorded rows inside ``[t0, t1]``, read from disk.
+
+        This is the interval read: it seeks to the nearest checkpoint in the
+        index and scans forward, so asking for ten seconds of an hour-long
+        recording costs ten seconds of parsing, not an hour. ``max_points``
+        decimates *while* reading, so the interval never has to exist in memory
+        in full.
+        """
+        start, stop = self.index().row_range(t0, t1)
+        return self.index().read_rows(
+            start, stop, max_points=max_points, unit=unit
+        )
+
+    def window_all(self, *, unit: str = "raw", max_points: int | None = None) -> ForceWindow:
+        """The whole recording as one window, decimated while streaming."""
+        return self.window(
+            *self.index().extent(), unit=unit, max_points=max_points
+        )
+
+    def overview(self, *, unit: str = "raw", buckets: int = DEFAULT_BUCKETS) -> Overview:
+        """A whole-session envelope, cached per (unit, bucket count)."""
+        key = (str(unit), int(buckets))
+        if key not in self._overviews:
+            self._overviews[key] = self.index().overview(
+                unit=str(unit), buckets=int(buckets)
+            )
+        return self._overviews[key]
+
+    def statistics(self, t0: float, t1: float, *, unit: str = "raw"):
+        """Statistics of an interval, computed from the interval's own rows."""
+        return self.window(t0, t1, unit=unit).statistics(unit)
+
     # ------------------------------------------------------------------ policy
     def depth_policy(self) -> dict:
         return dict(self.info().session.get("depth") or {})
@@ -372,6 +531,8 @@ class RecordingReader:
             warnings.append("rows_skipped")
         if self.info().frames_dropped:
             warnings.append("frames_dropped")
+        if not self.info().frames_out_of_order or not self.info().forces_out_of_order:
+            warnings.append("out_of_order")
         return warnings
 
     def iter_forces(self, batch: int = 4096) -> Iterator[tuple[np.ndarray, np.ndarray]]:
@@ -400,8 +561,56 @@ class RecordingReader:
             yield np.asarray(times), np.asarray(values).reshape(-1, 6)
 
 
+def _is_time_ordered(values) -> bool:
+    """True when ``values`` never decreases (an empty or single value is ordered)."""
+    previous = None
+    for value in values:
+        if previous is not None and float(value) < float(previous):
+            return False
+        previous = value
+    return True
+
+
+def decimate_window(window: ForceWindow, max_points: int, *, unit: str = "raw") -> ForceWindow:
+    """Reduce a window to about ``max_points``, keeping every axis' extremes.
+
+    The same rule the live store uses: each bucket contributes the extreme row of
+    **every plotted axis** in the active unit, so a shear spike survives a flat
+    normal channel. The result is marked ``decimated``.
+    """
+    total = len(window)
+    max_points = max(4, int(max_points))
+    if total <= max_points:
+        return window
+    values = window.value_for_unit(unit)
+    def index_of(row: int) -> int:
+        return int(window.frame_ids[row]) if total else row
+    bucket = max(1, total // max(1, max_points // 6))
+    picked: set[int] = set()
+    for start in range(0, total, bucket):
+        stop = min(total, start + bucket)
+        for axis in range(3):
+            column = values[start:stop, axis]
+            finite = np.isfinite(column)
+            if not finite.any():
+                continue
+            ordered = np.where(finite, column, -np.inf)
+            picked.add(start + int(np.argmin(ordered)))
+            picked.add(start + int(np.argmax(ordered)))
+    if not picked:
+        return window
+    rows = sorted(picked)
+    return ForceWindow(
+        times=window.times[rows],
+        raw=window.raw[rows],
+        newtons=window.newtons[rows],
+        valid=window.valid[rows],
+        frame_ids=window.frame_ids[rows],
+        decimated=True,
+    )
+
+
 def list_recordings(root: str | Path) -> list[Path]:
-    """Every recording directory under ``root``, newest first."""
     base = Path(root)
     if not base.is_dir():
         return []

shared/src/orisys_app_shared/record/sample.py

@@ -140,11 +140,16 @@ def to_record_sample(packet, *, calibration_version: str = "") -> RecordSample:
     valid = bool(getattr(packet, "valid", True))
     reason = str(getattr(packet, "invalid_reason", "") or "")
     stale = bool(getattr(packet, "stale_forces", False))
-    # The packet's own version wins: it was stamped by the acquisition thread at
-    # the moment of acquisition, so it is the only value that stays true when a
-    # calibration is applied in the middle of a session.
-    version = str(
-        getattr(packet, "calibration_version", "") or calibration_version or ""
+    # A packet that *carries* a version owns it, including the empty one: the
+    # acquisition thread stamps every frame, and an empty stamp is the device
+    # saying "no calibration loaded". Falling back to the session default there
+    # would resurrect coefficients that are no longer in effect. A packet from an
+    # older producer that has no version field at all still gets the default.
+    packet_version = getattr(packet, "calibration_version", None)
+    version = (
+        str(calibration_version or "")
+        if packet_version is None
+        else str(packet_version)
     )
     axes = calibrated_axes_of(packet)
 
@@ -214,3 +219,20 @@ def plottable_mask(values: Iterable[float], valid: Iterable[bool]) -> list[bool]
         bool(is_valid) and math.isfinite(float(value))
         for value, is_valid in zip(values, valid)
     ]
+
+
+def plottable_axis_mask(values, valid) -> "np.ndarray":
+    """``(n, axes)`` mask: one column per axis, decided for that axis alone.
+
+    The rule has to be per axis. A single-axis calibration is legitimate — the
+    device may have a normal-force curve and no shear curves — and an all-axes
+    rule hid a perfectly good FN because FX and FY had no newton value.
+    """
+    import numpy as np
+
+    block = np.asarray(values, dtype=np.float64).reshape(-1, 3)
+    validity = np.asarray(valid, dtype=bool).reshape(-1)
+    finite = np.isfinite(block)
+    if validity.size != block.shape[0]:
+        return finite
+    return finite & validity[:, None]

shared/src/orisys_app_shared/record/video_writer.py

@@ -0,0 +1,97 @@
+"""The video side of a recording: one rolling MP4 segment at a time.
+
+Kept apart from the CSV writer on purpose. The two have different failure modes —
+a full disk breaks both, but a codec that refuses to open only breaks this one —
+and the recorder's byte budget is about *these* frames, not about the rows.
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+from pathlib import Path
+from typing import Optional
+
+import cv2
+
+logger = logging.getLogger(__name__)
+
+
+class VideoSegments:
+    """Writes raw frames into ``raw_NNNN.mp4`` files, rolling on a time budget."""
+
+    def __init__(self, directory: Path, *, codec: str, fps: float, segment_seconds: float):
+        self.directory = Path(directory)
+        self.codec = str(codec)
+        self.fps = max(1.0, float(fps))
+        self.segment_seconds = float(segment_seconds)
+        self.segments = 0
+        self.video_frames = 0
+        self.frame_shape: Optional[tuple[int, int]] = None
+        self._writer: Optional[cv2.VideoWriter] = None
+        self._index = 0
+        self._frames_in_segment = 0
+        self._started = 0.0
+
+    # ------------------------------------------------------------------ writing
+    def write(self, frame) -> tuple[int, int]:
+        """Append one frame. Returns ``(segment, frame_in_segment)``."""
+        if self.frame_shape is None:
+            self.frame_shape = (int(frame.shape[1]), int(frame.shape[0]))
+        if self._should_roll():
+            self._close_segment()
+        if self._writer is None:
+            self._open_segment()
+        self._writer.write(frame)
+        self._frames_in_segment += 1
+        self.video_frames += 1
+        return (self._index, self._frames_in_segment)
+
+    def write_sequence(self, frames) -> int:
+        """Write a run of frames as one standalone segment. Returns the count."""
+        frames = [frame for frame in frames if frame is not None]
+        if not frames:
+            return 0
+        self.frame_shape = (int(frames[0].shape[1]), int(frames[0].shape[0]))
+        name = "preroll_0001.mp4"
+        writer = self._open_writer(self.directory / name)
+        written = 0
+        try:
+            for frame in frames:
+                writer.write(frame)
+                written += 1
+        finally:
+            writer.release()
+        return written
+
+    # ----------------------------------------------------------------- segments
+    def _should_roll(self) -> bool:
+        if self._writer is None:
+            return False
+        if self.segment_seconds <= 0:
+            return False
+        return (time.perf_counter() - self._started) >= self.segment_seconds
+
+    def _open_writer(self, path: Path) -> cv2.VideoWriter:
+        fourcc = cv2.VideoWriter_fourcc(*self.codec)
+        writer = cv2.VideoWriter(str(path), fourcc, self.fps, self.frame_shape)
+        if not writer.isOpened():
+            raise OSError(f"could not open video writer for {path.name}")
+        return writer
+
+    def _open_segment(self) -> None:
+        self._index += 1
+        path = self.directory / f"raw_{self._index:04d}.mp4"
+        self._writer = self._open_writer(path)
+        self._frames_in_segment = 0
+        self._started = time.perf_counter()
+        self.segments = self._index
+
+    def _close_segment(self) -> None:
+        if self._writer is None:
+            return
+        self._writer.release()
+        self._writer = None
+
+    def close(self) -> None:
+        self._close_segment()

shared/tests/test_calibration_identity.py

@@ -0,0 +1,280 @@
+"""Calibration identity and operator events as the acquisition thread sees them.
+
+The review's P1-7: the application event was written before the device had loaded
+anything, the recorded ``calibration_id`` never changed mid-session, and an
+uncalibrated device could keep claiming the coefficients of an earlier session.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from orisys_app_shared.acquisition.sdk_worker import SdkAcquisitionWorker
+from orisys_app_shared.record import ExperimentRecorder, RecordConfig
+from orisys_app_shared.record.calibration_snapshot import (
+    list_calibration_snapshots,
+    snapshot_name,
+    store_calibration_snapshot,
+)
+
+SCRATCH = Path(__file__).resolve().parent / "_scratch"
+
+
+def _root() -> Path:
+    shutil.rmtree(SCRATCH, ignore_errors=True)
+    SCRATCH.mkdir(parents=True, exist_ok=True)
+    return SCRATCH
+
+
+class _Sensor:
+    """A device with a swappable calibration, like the SDK's Sensor."""
+
+    def __init__(self, calibration=None, path=None):
+        self.calibration = calibration
+        self.calibration_path = path
+        self.applied: list[str] = []
+        self.resets = 0
+
+    def set_calibration(self, path):
+        if str(path) == "reject-me":
+            raise ValueError("bad document")
+        self.calibration = object()
+        self.calibration_path = Path(path)
+        self.applied.append(str(path))
+
+    def reset(self):
+        self.resets += 1
+
+
+def _calibration_file(root: Path, name: str, body: str = '{"unit": "N"}') -> Path:
+    path = root / name
+    path.write_text(body, encoding="utf-8")
+    return path
+
+
+def _worker(**kwargs) -> SdkAcquisitionWorker:
+    for key in ("on_calibration_applied", "on_reset_applied"):
+        kwargs.setdefault(key, None)
+    return SdkAcquisitionWorker(**kwargs)
+
+
+# ------------------------------------------------------------------ worker state
+
+
+def test_a_new_session_does_not_inherit_the_previous_version():
+    """An uncalibrated device must report no calibration, not the last one."""
+    root = _root()
+    worker = _worker()
+    worker._calibration_version = "previous-session-hash"
+    worker._stop.set()
+    worker.start("clip.mp4", config_ref=str(_calibration_file(root, "sensor.json")))
+    assert worker._calibration_version == "", (
+        "the new session kept the previous session's calibration version"
+    )
+    worker.stop(wait=True)
+
+
+def test_the_applied_version_is_the_content_hash_and_carries_the_frame():
+    root = _root()
+    applied: list[tuple] = []
+    worker = _worker(on_calibration_applied=lambda *args: applied.append(args))
+    path = _calibration_file(root, "force_calibration.json")
+    sensor = _Sensor()
+    worker._session_id = 4
+    worker.set_pending_calibration(path)
+    worker._consume_pending_calibration(sensor, frame_id=17, mono_t=0.5)
+
+    assert len(applied) == 1
+    got_path, version, frame_id, mono_t, session = applied[0]
+    assert got_path == str(path)
+    assert version and version == worker._calibration_version
+    assert version != str(path), "the path was used as the version"
+    assert (frame_id, mono_t, session) == (17, 0.5, 4)
+    assert sensor.applied == [str(path)]
+
+
+def test_a_rejected_calibration_keeps_the_version_the_device_still_has():
+    root = _root()
+    applied: list[tuple] = []
+    worker = _worker(on_calibration_applied=lambda *args: applied.append(args))
+    sensor = _Sensor()
+    worker._session_id = 1
+    worker._calibration_version = "loaded-earlier"
+    worker.set_pending_calibration("reject-me")
+    with pytest.raises(ValueError):
+        worker._consume_pending_calibration(sensor, frame_id=3, mono_t=0.1)
+    assert not applied, "a rejected document was reported as applied"
+    assert worker._calibration_version == "loaded-earlier", (
+        "the device still has those coefficients, so the version is still true"
+    )
+
+
+def test_a_zeroing_is_applied_by_the_worker_and_reported_with_its_frame():
+    root = _root()
+    events: list[tuple] = []
+    worker = _worker(on_reset_applied=lambda *args: events.append(args))
+    sensor = _Sensor()
+    worker._session_id = 2
+    worker.reset()  # a request, not the zeroing itself
+    assert sensor.resets == 0, "the zeroing ran on the caller's thread"
+    worker._consume_pending_reset(sensor, frame_id=42, mono_t=1.4)
+    assert sensor.resets == 1
+    assert events == [(42, 1.4, 2)]
+    # The request is consumed once.
+    worker._consume_pending_reset(sensor, frame_id=43, mono_t=1.5)
+    assert events == [(42, 1.4, 2)]
+
+
+def test_a_started_session_drops_a_stale_zeroing_request():
+    root = _root()
+    events: list[tuple] = []
+    worker = _worker(on_reset_applied=lambda *args: events.append(args))
+    worker._stop.set()
+    worker.start("clip.mp4", config_ref=str(_calibration_file(root, "sensor.json")))
+    sensor = _Sensor()
+    worker._consume_pending_reset(sensor, frame_id=1, mono_t=0.0)
+    assert sensor.resets == 0, "a reset requested before this session fired into it"
+    assert not events
+    worker.stop(wait=True)
+
+
+# ------------------------------------------------------------------- snapshots
+
+
+def test_a_snapshot_is_content_addressed_and_listed():
+    root = _root()
+    directory = root / "session"
+    source = _calibration_file(root, "force_calibration_box.json", '{"a": 1}')
+    name = store_calibration_snapshot(directory, source, "abcdef1234567890")
+    assert name == "calibration/force_calibration_abcdef123456.json"
+    assert (directory / name).is_file()
+    assert (directory / name).read_text(encoding="utf-8") == '{"a": 1}'
+    # Applying the same coefficients twice stores one file.
+    again = store_calibration_snapshot(directory, source, "abcdef1234567890")
+    assert again == name
+    assert list_calibration_snapshots(directory) == [name]
+    # A different calibration cannot collide with it.
+    other = _calibration_file(root, "force_calibration_box.json", '{"a": 2}')
+    second = store_calibration_snapshot(directory, other, "ffffff000000")
+    assert second != name
+    assert len(list_calibration_snapshots(directory)) == 2
+    assert snapshot_name("", source).startswith("force_calibration_")
+
+
+def test_the_recording_metadata_names_the_coefficients_it_ran_with():
+    root = _root()
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    directory = recorder.start(root=root, session_id=1, metadata={})
+    source = _calibration_file(root, "force_calibration_box.json", '{"a": 1}')
+    name = store_calibration_snapshot(directory, source, "abcdef1234567890")
+    recorder.note_calibration_snapshot(
+        version="abcdef1234567890", name=name, frame_id=12, mono_t=0.4
+    )
+    recorder.log_event(
+        "calibration_applied", name, mono_t=0.4, frame_id=12
+    )
+    recorder.stop()
+
+    meta = json.loads((Path(directory) / "session.json").read_text(encoding="utf-8"))
+    snapshots = meta["calibration_snapshots"]
+    assert snapshots and snapshots[0]["version"] == "abcdef1234567890"
+    assert snapshots[0]["file"] == name
+    assert snapshots[0]["frame_id"] == 12
+    assert snapshots[0]["mono_t"] == pytest.approx(0.4)
+    event = next(e for e in meta["events"] if e["kind"] == "calibration_applied")
+    assert event["frame_id"] == 12
+    assert event["mono_t"] == pytest.approx(0.4), "the event lost its acquisition time"
+    assert event["at"], "the event lost its wall clock"
+
+    # The CSV carries the same acquisition time in the session's relative base.
+    # An event that predates the first frame has no base yet, so its relative
+    # time is empty — the absolute (session-relative) time is in the metadata.
+    with (Path(directory) / "events.csv").open(encoding="utf-8", newline="") as handle:
+        import csv as _csv
+
+        rows = list(_csv.DictReader(handle))
+    applied = next(row for row in rows if row["kind"] == "calibration_applied")
+    assert applied["frame_id"] == "12"
+    assert applied["t_rel_s"] == "", "an event before the first frame invented a base"
+
+
+def test_an_event_during_the_recording_gets_the_session_relative_time():
+    import csv as _csv
+
+    from orisys_app_shared.acquisition.sdk_worker import SdkPacket
+
+    root = _root()
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    directory = recorder.start(root=root, session_id=1, metadata={})
+    frame = np.zeros((4, 4, 3), np.uint8)
+    recorder.on_packet(
+        SdkPacket(
+            raw_frame=frame,
+            processed_frame=frame,
+            timestamp=1000.0,
+            mono_t=0.5,
+            fps=30.0,
+            process_ms=1.0,
+            frame_id=7,
+        )
+    )
+    recorder.log_event("zeroing", "tracker reset", mono_t=0.9, frame_id=9)
+    recorder.stop()
+
+    with (Path(directory) / "events.csv").open(encoding="utf-8", newline="") as handle:
+        rows = list(_csv.DictReader(handle))
+    zeroing = next(row for row in rows if row["kind"] == "zeroing")
+    assert float(zeroing["t_rel_s"]) == pytest.approx(0.4, abs=1e-6)
+    assert zeroing["frame_id"] == "9"
+
+
+def test_an_event_without_a_mono_time_still_gets_a_wall_clock():
+    root = _root()
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    directory = recorder.start(root=root, session_id=1, metadata={})
+    recorder.log_event("operator_note", "no frame attached")
+    recorder.stop()
+    meta = json.loads((Path(directory) / "session.json").read_text(encoding="utf-8"))
+    event = next(e for e in meta["events"] if e["kind"] == "operator_note")
+    assert event["at"], "the event lost its wall clock"
+    assert event["frame_id"] == -1
+
+
+def test_the_recorded_row_uses_the_packet_version_not_the_session_default():
+    from orisys_app_shared.record.sample import to_record_sample
+
+    class Packet:
+        frame_id = 3
+        mono_t = 0.2
+        wall_t = 0.2
+        valid = True
+        calibration_version = ""
+        calibrated_axes = ("fn",)
+        force_calibrated = True
+        raw_available = True
+
+    # An empty packet version is the device saying "no calibration": the session
+    # default must not be pasted in behind its back.
+    assert to_record_sample(Packet(), calibration_version="session-default").calibration_version == ""
+
+    class LegacyPacket(Packet):
+        calibration_version = None  # an older producer without the field
+
+    # ...but a packet that has no version field at all still gets the default.
+    assert (
+        to_record_sample(LegacyPacket(), calibration_version="session-default").calibration_version
+        == "session-default"
+    )
+
+    class StampedPacket(Packet):
+        calibration_version = "packet-hash"
+
+    assert (
+        to_record_sample(StampedPacket(), calibration_version="session-default").calibration_version
+        == "packet-hash"
+    )

shared/tests/test_calibration_model.py

@@ -8,6 +8,9 @@ model.
 
 from __future__ import annotations
 
+import shutil
+from pathlib import Path
+
 import pytest
 
 from orisys_app_shared.force.calibration_io import (
@@ -17,10 +20,19 @@ from orisys_app_shared.force.calibration_io import (
     build_force_calibration_v2,
     merge_samples,
     predict_channel,
+    write_force_calibration,
 )
 from orisys_app_shared.force.calibration_recorder import CalibrationRecorder
 from orisys_app_shared.force.session import SessionSample
 
+SCRATCH = Path(__file__).resolve().parent / "_scratch"
+
+
+def _root() -> Path:
+    shutil.rmtree(SCRATCH, ignore_errors=True)
+    SCRATCH.mkdir(parents=True, exist_ok=True)
+    return SCRATCH
+
 
 def _point(recorder, load: float, raw: float, *, channel: str = "fn"):
     recorder.start(channel=channel, reference_load_n=load, duration_s=0.05)
@@ -151,3 +163,84 @@ def test_a_channel_with_one_point_is_not_exported():
     store.aggregate("fn", CalSample(weight_n=1.0, raw_mean=10.0, n=10))
     with pytest.raises(ValueError):
         build_force_calibration_v2(store)
+
+
+# ---------------------------------------------------------------- one preparation
+
+
+def _session_with_duplicates_and_validation():
+    recorder = CalibrationRecorder()
+    points = [
+        _point(recorder, 1.0, 100.0),
+        _point(recorder, 2.0, 200.0),
+        # A deliberate repeat of load 2.0 at a different raw level.
+        _point(recorder, 2.0, 240.0),
+        _point(recorder, 4.0, 400.0),
+        _point(recorder, 3.0, 290.0),
+    ]
+    recorder.set_validation(points[4], True)
+    return recorder, points
+
+
+def test_duplicate_loads_are_merged_before_the_model_is_built():
+    _recorder, _points = _session_with_duplicates_and_validation()
+    prepared = _recorder.prepared()
+    assert prepared.samples["fn"], "no training samples"
+    loads = [sample.weight_n for sample in prepared.samples["fn"]]
+    assert loads == sorted(set(loads)), f"duplicate loads were not merged: {loads}"
+    assert len(loads) == 3
+    merged = next(sample for sample in prepared.samples["fn"] if sample.weight_n == 2.0)
+    assert merged.n == 16, "the repeat's samples were not combined"
+    assert merged.raw_mean == pytest.approx((200.0 + 240.0) / 2.0)
+    assert prepared.points_in_fit("fn") == 3
+
+
+def test_the_exported_file_predicts_exactly_what_the_panel_shows():
+    """Read the written document back and compare it with the diagnostics.
+
+    The panel's numbers must describe the file that is actually written, so the
+    test loads the file with the SDK (not with the prepared object) and predicts
+    every training and validation point from it.
+    """
+    import orisys.calibration as sdk_calibration
+
+    recorder, points = _session_with_duplicates_and_validation()
+    prepared = recorder.prepared(meta={"sensor_type": "box", "sn": "SN9"})
+    assert prepared.ok
+
+    document = prepared.to_document()
+    path = _root() / "force_calibration.json"
+    write_force_calibration(document, path)
+    from_file = sdk_calibration.load(path, strict=True)
+
+    for point in points:
+        raw = float(point.loaded_stats().mean)
+        predicted = float(from_file.get("fn").convert(raw))
+        value = prepared.predict("fn", raw)
+        assert predicted == pytest.approx(value, abs=1e-9)
+        if point.is_validation:
+            assert prepared.validation_errors[id(point)] == pytest.approx(
+                abs(predicted - point.reference_load_n), abs=1e-9
+            )
+        else:
+            assert prepared.residuals[id(point)] == pytest.approx(
+                predicted - point.reference_load_n, abs=1e-9
+            )
+
+    # The validation point is measured, never built in.
+    assert prepared.validation_points("fn") == 1
+    assert prepared.max_validation_error("fn") > 0.0
+    exported_raws = sorted(point[0] for point in document["channels"]["normal"]["points"])
+    assert exported_raws == [100.0, 220.0, 400.0], exported_raws
+
+
+def test_the_recorder_fit_and_the_preparation_agree():
+    recorder, _points = _session_with_duplicates_and_validation()
+    prepared = recorder.prepared()
+    fit = recorder.fit("fn", prepared=prepared)
+    assert fit.n_points == prepared.points_in_fit("fn")
+    assert fit.breakpoints == prepared.breakpoints("fn")
+    assert fit.max_residual_n == pytest.approx(prepared.max_residual("fn"))
+    assert fit.validation_error_n == pytest.approx(prepared.max_validation_error("fn"))
+    # ...and a fit that prepares on its own gets the same numbers.
+    assert recorder.fit("fn").as_dict() == fit.as_dict()

shared/tests/test_calibration_record.py

@@ -31,15 +31,15 @@ def _root() -> Path:
     return SCRATCH
 
 
-def _recorded_point(index_offset: float = 0.0):
+def _recorded_point(load: float = 2.0, offset: float = 0.0):
     recorder = CalibrationRecorder()
-    recorder.start(channel="fn", reference_load_n=2.0, duration_s=0.05)
+    recorder.start(channel="fn", reference_load_n=load, duration_s=0.05)
     for index in range(6):
         recorder.feed(
             SessionSample(
                 index,
                 index * 1.0 / 30.0,
-                fn_raw=float(index) + index_offset,
+                fn_raw=float(index) + offset,
                 fx_raw=1.0,
                 fy_raw=2.0,
                 fn_n=float(index) * 10.0,
@@ -126,9 +126,9 @@ def test_a_newer_format_is_loaded_with_a_warning():
 
 def test_the_recorder_can_reopen_a_saved_session():
     points = [
-        _recorded_point(),
-        _recorded_point(5.0),
-        _recorded_point(9.0),
+        _recorded_point(1.0),
+        _recorded_point(2.0, 5.0),
+        _recorded_point(3.0, 9.0),
     ]
     path = write_calibration_session(_root() / "session.json", points)
     restored = read_calibration_session(path).points
@@ -141,3 +141,76 @@ def test_the_recorder_can_reopen_a_saved_session():
     fit = recorder.fit("fn")
     assert fit.validation_points == 1
     assert fit.n_points == 2
+
+
+def test_reopened_points_at_one_load_are_one_calibrated_load():
+    """Three points at the same reference load are one load, not a curve.
+
+    They are merged (n-weighted) before the build, so a session that only ever
+    sampled one load cannot produce a model — which is the honest answer.
+    """
+    recorder = CalibrationRecorder()
+    recorder.load_points(
+        [_recorded_point(2.0), _recorded_point(2.0, 5.0), _recorded_point(2.0, 9.0)],
+        replace=True,
+    )
+    prepared = recorder.prepared()
+    assert len(prepared.training) == 3
+    assert len(prepared.samples["fn"]) == 1
+    assert prepared.samples["fn"][0].n == 15, "the valid counts were not summed"
+    assert prepared.model is None, "a single load produced a model"
+    assert recorder.fit("fn").n_points == 1
+
+
+# ------------------------------------------------------- reopened diagnostics
+
+
+def test_a_reopened_record_keeps_its_device_identity():
+    """Device A's record stays A's, whatever is connected now."""
+    points = [_recorded_point(1.0), _recorded_point(2.0, 5.0)]
+    path = write_calibration_session(
+        _root() / "session.json",
+        points,
+        meta={
+            "sensor_type": "box",
+            "sn": "SN-A",
+            "config_ref": "D:/lab/deviceA/sensor.json",
+        },
+    )
+    record = read_calibration_session(path)
+    identity = record.identity()
+    assert identity["sensor_type"] == "box"
+    assert identity["sn"] == "SN-A"
+    assert identity["config_ref"] == "D:/lab/deviceA/sensor.json"
+    assert identity["created_at"] == record.created_at
+    assert "SN-A" in record.describe()
+
+
+def test_the_three_axis_trace_is_rebuilt_from_the_samples():
+    """A record whose trace was never written must still be inspectable."""
+    import json
+
+    point = _recorded_point(1.0)
+    document = build_session_document([point])
+    # Simulate a document that only kept the audit series (or an older writer
+    # that stored a truncated trace).
+    document["points"][0]["trace"] = {}
+    document["points"][0]["duration_s"] = 0.0
+    path = _root() / "session.json"
+    path.write_text(json.dumps(document), encoding="utf-8")
+
+    restored = read_calibration_session(path).points[0]
+    assert set(restored.trace) == {"fn", "fx", "fy"}, restored.trace
+    assert len(restored.trace["fn"]) == 5  # the valid samples
+    assert restored.trace_dt_s == pytest.approx(1.0 / 30.0, abs=1e-6)
+    assert restored.duration_s == pytest.approx(5.0 / 30.0, abs=1e-6)
+
+
+def test_a_stored_trace_is_not_overwritten_on_reopen():
+    point = _recorded_point(1.0)
+    document = build_session_document([point])
+    document["points"][0]["trace"] = {"fn": [1.0, 2.0]}
+    path = _root() / "session.json"
+    path.write_text(json.dumps(document), encoding="utf-8")
+    restored = read_calibration_session(path).points[0]
+    assert restored.trace["fn"] == [1.0, 2.0]

shared/tests/test_depth_replay.py

@@ -16,7 +16,12 @@ from unittest.mock import patch
 import numpy as np
 import pytest
 
-from orisys_app_shared.record.depth import DepthRecomputer
+from orisys_app_shared.record.depth import (
+    STATE_APPROXIMATE,
+    STATE_EXACT,
+    STATE_PREROLL,
+    DepthRecomputer,
+)
 
 SCRATCH = Path(__file__).resolve().parent / "_scratch"
 
@@ -36,6 +41,12 @@ class _CountingSensor:
         _CountingSensor.opened.append({"args": args, "kwargs": kwargs})
         self.n = 0
         self.chain_calls = 0
+        self.initial_img = "reference-frame"
+        # The real Sensor opens its source in the constructor, so the first
+        # source is part of its history too.
+        self.vid_src = str(args[0]) if args else ""
+        self.sources: list[str] = [self.vid_src] if self.vid_src else []
+        self.resets = 0
         self.info = SimpleNamespace(DEPTH=1)
 
     def get_img(self):
@@ -48,9 +59,17 @@ class _CountingSensor:
     def compute_contact(self):
         self.chain_calls += 1
 
+    def reset(self):
+        self.resets += 1
+
     def read_info(self, _channel):
         return np.array([self.n])
 
+    def _open_video_source(self, _override, _box):
+        self.sources.append(str(self.vid_src))
+        # A real reopen restarts the file from its first frame.
+        self.n = 0
+
     def disconnect(self):
         return None
 
@@ -116,13 +135,18 @@ def test_stepping_continues_from_the_current_position():
     computer.close()
 
 
-def test_stepping_backwards_reopens_the_segment():
+def test_stepping_backwards_reopens_the_source_but_keeps_the_processor():
     _CountingSensor.opened.clear()
     computer = DepthRecomputer(_reader(_session("/cfg/sensor.json")))
     with patch("orisys.Sensor", _CountingSensor):
         assert computer._depth_from_segment(Path("raw_0001.mp4"), 5).tolist() == [5]
+        # Reading an earlier frame restarts the file; the processor object (and
+        # therefore the tracker) is not thrown away.
         assert computer._depth_from_segment(Path("raw_0001.mp4"), 2).tolist() == [2]
-    assert len(_CountingSensor.opened) == 2, "seeking backwards needs a fresh source"
+        sensor = computer._sensor._sensor
+    assert len(_CountingSensor.opened) == 1, "the processor was recreated"
+    assert len(sensor.sources) == 2
+    assert all(source.endswith("raw_0001.mp4") for source in sensor.sources)
     computer.close()
 
 
@@ -165,11 +189,44 @@ def test_the_plan_starts_at_the_last_reset_in_the_same_segment():
     computer = DepthRecomputer(reader)
     plan = computer.plan_for(frames[8])
     assert plan.stop_at == 9
+    assert plan.anchor == "reset"
+    assert plan.state == STATE_EXACT
+    assert plan.reset_frame_id == 6
+    assert len(plan.steps) == 1
+    step = plan.steps[0]
+    assert step.source == reader.directory / "raw_0001.mp4"
+    assert step.to_frame == 9
+    assert step.reset_after == 7, "the reset boundary is not re-applied by the plan"
+    assert plan.frames_to_read == 9
     assert plan.preroll is None
     assert plan.warnings == ()
     computer.close()
 
 
+def test_a_reset_that_is_not_in_the_video_is_not_a_recovered_state():
+    """Finding the event is not the same as being able to start there."""
+    frames = [
+        SimpleNamespace(
+            frame_id=index,
+            t_rel_s=index / 30.0,
+            segment=1,
+            segment_frame=index + 1,
+            in_video=index != 6,  # the reset frame never reached the video
+            dropped=False,
+        )
+        for index in range(10)
+    ]
+    reader = _reader(_session("/cfg/sensor.json"), frames=frames)
+    reader.last_reset_frame = lambda frame_id: 6
+    computer = DepthRecomputer(reader)
+    plan = computer.plan_for(frames[8])
+    assert plan.anchor != "reset"
+    assert plan.state == STATE_APPROXIMATE
+    assert "reset_frame_not_in_video" in plan.warnings
+    assert "state_approximate" in plan.warnings
+    computer.close()
+
+
 def test_the_plan_uses_the_pre_roll_for_the_start_of_a_recording():
     frames = [
         SimpleNamespace(
@@ -195,6 +252,8 @@ def test_the_plan_uses_the_pre_roll_for_the_start_of_a_recording():
     plan = computer.plan_for(frames[3])
     assert plan.preroll is not None
     assert plan.preroll_frames == 30
+    assert plan.anchor == "preroll"
+    assert plan.state == STATE_PREROLL
     assert plan.warnings == ()
     computer.close()
 
@@ -215,10 +274,127 @@ def test_a_segment_without_history_is_flagged_approximate():
     reader = _reader(_session("/cfg/sensor.json"), frames=frames)
     computer = DepthRecomputer(reader)
     plan = computer.plan_for(frames[2])
+    assert plan.state == STATE_APPROXIMATE
+    assert plan.anchor == "segment_start"
     assert "state_approximate" in plan.warnings
     computer.close()
 
 
+def test_a_reset_in_an_earlier_segment_is_carried_across_by_swapping_the_source():
+    """The processor survives the swap, so the state *can* cross a segment."""
+    frames = [
+        SimpleNamespace(
+            frame_id=index,
+            t_rel_s=index / 30.0,
+            segment=1 if index < 3 else 2,
+            segment_frame=index + 1 if index < 3 else index - 2,
+            in_video=True,
+            dropped=False,
+        )
+        for index in range(6)
+    ]
+    reader = _reader(
+        _session("/cfg/sensor.json"),
+        segments=("raw_0001.mp4", "raw_0002.mp4"),
+        frames=frames,
+    )
+    reader.last_reset_frame = lambda frame_id: 1
+    computer = DepthRecomputer(reader)
+    plan = computer.plan_for(frames[5])
+    assert plan.anchor == "reset"
+    assert plan.state == STATE_EXACT
+    assert "state_across_segment" not in plan.warnings
+    assert [step.source.name for step in plan.steps] == ["raw_0001.mp4", "raw_0002.mp4"]
+    assert plan.steps[0].reset_after == 2
+    assert plan.steps[1].to_frame == 3
+    computer.close()
+
+
+def test_a_replay_that_would_read_too_far_back_is_reported_not_attempted():
+    frames = [
+        SimpleNamespace(
+            frame_id=index,
+            t_rel_s=index / 30.0,
+            segment=1,
+            segment_frame=index + 1,
+            in_video=True,
+            dropped=False,
+        )
+        for index in range(20)
+    ]
+    reader = _reader(_session("/cfg/sensor.json"), frames=frames)
+    reader.last_reset_frame = lambda frame_id: 1
+    computer = DepthRecomputer(reader)
+    with patch("orisys_app_shared.record.depth.MAX_REPLAY_FRAMES", 5):
+        plan = computer.plan_for(frames[10])
+    assert "state_too_far_back" in plan.warnings
+    assert plan.state == STATE_APPROXIMATE
+    computer.close()
+
+
+def test_the_same_frame_is_answered_from_the_cache():
+    """A second request for the frame just read is a cache hit, not None."""
+    _CountingSensor.opened.clear()
+    computer = DepthRecomputer(_reader(_session("/cfg/sensor.json")))
+    with patch("orisys.Sensor", _CountingSensor):
+        first = computer._depth_from_segment(Path("raw_0001.mp4"), 3)
+        second = computer._depth_from_segment(Path("raw_0001.mp4"), 3)
+        sensor = computer._sensor._sensor
+    assert first.tolist() == [3]
+    assert second is not None and second.tolist() == [3]
+    assert sensor.n == 3, "the cached answer re-read the stream"
+    computer.close()
+
+
+def test_the_processor_survives_a_pre_roll_to_segment_switch():
+    """Only the video source is replaced; the tracker state must survive."""
+    _CountingSensor.opened.clear()
+    reader = _reader(_session("/cfg/sensor.json"))
+    preroll = reader.directory / "preroll_0001.mp4"
+    preroll.write_bytes(b"")
+    computer = DepthRecomputer(reader)
+    with patch("orisys.Sensor", _CountingSensor):
+        computer._feed(preroll, 2)
+        sensor = computer._sensor._sensor
+        assert sensor.n == 2, "the pre-roll frames were not fed"
+        # Feed the segment: same processor, new source, position restarted.
+        computer._feed(reader.directory / "raw_0001.mp4", 1)
+    assert len(_CountingSensor.opened) == 1, "the processor was recreated"
+    assert computer._sensor._sensor is sensor
+    assert len(sensor.sources) == 2
+    assert sensor.sources[0].endswith("preroll_0001.mp4")
+    assert sensor.sources[1].endswith("raw_0001.mp4")
+    assert sensor.initial_img == "reference-frame", "the tracker reference was reset"
+    computer.close()
+
+
+def test_a_reset_boundary_is_re_applied_during_the_replay():
+    """The live zeroing cleared the tracker; the replay must do the same."""
+    _CountingSensor.opened.clear()
+    frames = [
+        SimpleNamespace(
+            frame_id=index,
+            t_rel_s=index / 30.0,
+            segment=1,
+            segment_frame=index + 1,
+            in_video=True,
+            dropped=False,
+        )
+        for index in range(10)
+    ]
+    reader = _reader_with_config(frames)
+    reader.last_reset_frame = lambda frame_id: 6
+    reader.frame_at = lambda _t: frames[8]
+    computer = DepthRecomputer(reader)
+    with patch("orisys.Sensor", _CountingSensor):
+        result = computer.depth_at(frames[8].t_rel_s)
+        sensor = computer._sensor._sensor
+    assert result.ok, result.error
+    assert sensor.resets == 1, "the reset boundary was not re-applied"
+    assert result.warnings == (), result.warnings
+    computer.close()
+
+
 def test_a_dropped_frame_is_reported_not_substituted():
     """The index marks the hole; replay must say so instead of showing a neighbour."""
     frames = [

shared/tests/test_display_units.py

@@ -0,0 +1,139 @@
+"""Display unit selection and the per-axis drawing rule.
+
+Two defects are covered here:
+
+* the live charts chose their unit from the device-level ``force_calibrated``
+  flag, so a device with only its normal-force axis calibrated either showed
+  every axis in raw counts or showed raw counts inside newton panels;
+* the drawing rule asked for **all three** axes to be finite, so a valid FN
+  disappeared from the chart whenever FX/FY had no calibrated value.
+"""
+
+from __future__ import annotations
+
+import math
+
+import numpy as np
+import pytest
+
+from orisys_app_shared.acquisition.sdk_worker import SdkPacket
+from orisys_app_shared.force import select_display_forces
+from orisys_app_shared.force.session import SessionSample, SessionStore
+from orisys_app_shared.record.sample import to_record_sample
+
+NAN = float("nan")
+
+
+def _packet(**overrides) -> SdkPacket:
+    frame = np.zeros((4, 4, 3), np.uint8)
+    kwargs = dict(
+        raw_frame=frame,
+        processed_frame=frame,
+        timestamp=1000.0,
+        fps=30.0,
+        process_ms=1.0,
+        normal_force=1.5,
+        shear_x=0.25,
+        shear_y=-0.25,
+        normal_force_raw=123.0,
+        shear_x_raw=2.0,
+        shear_y_raw=-2.0,
+        force_calibrated=True,
+        calibrated_axes=("fn", "fx", "fy"),
+        frame_id=0,
+    )
+    kwargs.update(overrides)
+    return SdkPacket(**kwargs)
+
+
+def test_newton_mode_marks_uncalibrated_axes_unavailable():
+    sample = to_record_sample(_packet(calibrated_axes=("fn",)))
+    display = select_display_forces(units_pref="newton", sample=sample)
+    assert display.unit == "N"
+    assert display.values[0] == pytest.approx(1.5)
+    assert math.isnan(display.values[1]) and math.isnan(display.values[2])
+    assert display.available == (True, False, False)
+    assert display.unavailable_axes() == ("fx", "fy")
+    assert not display.using_raw_fallback
+
+
+def test_raw_mode_shows_the_raw_counts_for_every_axis():
+    sample = to_record_sample(_packet(calibrated_axes=("fn",)))
+    display = select_display_forces(units_pref="raw", sample=sample)
+    assert display.unit == "raw"
+    assert display.values == (123.0, 2.0, -2.0)
+    assert display.available == (True, True, True)
+    assert not display.using_raw_fallback
+
+
+def test_an_uncalibrated_device_falls_back_to_raw_and_says_so():
+    sample = to_record_sample(
+        _packet(force_calibrated=False, calibrated_axes=())
+    )
+    display = select_display_forces(units_pref="newton", sample=sample)
+    assert display.unit == "raw"
+    assert display.using_raw_fallback, "the fallback must be announceable"
+    assert display.requested_unit == "N"
+    assert display.values == (123.0, 2.0, -2.0)
+
+
+def test_a_failed_readout_has_no_available_axis():
+    sample = to_record_sample(
+        _packet(valid=False, invalid_reason="compute failed", stale_forces=True)
+    )
+    for unit in ("N", "raw"):
+        display = select_display_forces(units_pref=unit, sample=sample)
+        assert display.available == (False, False, False)
+        assert all(math.isnan(value) for value in display.values)
+
+
+def _store_with_one_calibrated_axis() -> SessionStore:
+    store = SessionStore()
+    for index in range(5):
+        store.append(
+            SessionSample(
+                index,
+                index / 30.0,
+                fn_raw=100.0 + index,
+                fx_raw=2.0,
+                fy_raw=-2.0,
+                fn_n=1.0 + index * 0.1,
+                fx_n=NAN,
+                fy_n=NAN,
+                valid=True,
+                calibrated_axes=("fn",),
+            )
+        )
+    return store
+
+
+def test_a_valid_axis_is_drawn_even_when_the_others_have_no_value():
+    """The all-axes rule hid a good FN because FX/FY were uncalibrated."""
+    slice_ = _store_with_one_calibrated_axis().data()
+    times, values, mask = slice_.plottable("N")
+    assert mask.shape == (5, 3)
+    assert mask[:, 0].all(), "the calibrated axis was hidden"
+    assert not mask[:, 1].any()
+    assert not mask[:, 2].any()
+
+    fn_times, fn_values = slice_.plottable_axis("N", 0)
+    assert fn_times.size == 5 == fn_values.size
+    assert float(fn_values.max()) == pytest.approx(1.4)
+    fx_times, _fx_values = slice_.plottable_axis("N", 1)
+    assert fx_times.size == 0
+
+
+def test_the_raw_triplet_is_still_complete_for_the_same_samples():
+    slice_ = _store_with_one_calibrated_axis().data()
+    for index in range(3):
+        times, values = slice_.plottable_axis("raw", index)
+        assert times.size == 5
+        assert np.isfinite(values).all()
+
+
+def test_an_invalid_sample_is_excluded_from_every_axis():
+    store = _store_with_one_calibrated_axis()
+    store.append(SessionSample(9, 9.0, fn_n=NAN, valid=False))
+    slice_ = store.data()
+    _times, _values, mask = slice_.plottable("N")
+    assert not mask[-1].any(), "an invalid sample reached a drawing path"

shared/tests/test_history_source.py

@@ -0,0 +1,218 @@
+"""Long-history browsing: interval reads, envelopes and one curve interface.
+
+The replay used to load every row of a recording before drawing anything, so a
+multi-hour session could not be shown at all. These tests pin the two things that
+make it possible — an interval read bounded by the interval, and a whole-session
+envelope built while streaming the file once — and that the charts can read from
+either a session store or a recording through the same interface.
+"""
+
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from orisys_app_shared.acquisition.sdk_worker import SdkPacket
+from orisys_app_shared.force.session import SessionSample, SessionStore
+from orisys_app_shared.force.source import RecordingCurveSource, StoreCurveSource
+from orisys_app_shared.record import ExperimentRecorder, RecordConfig, RecordingReader
+from orisys_app_shared.record.index import RecordingIndex
+
+SCRATCH = Path(__file__).resolve().parent / "_scratch_history"
+
+ROWS = 12_000
+SPIKE_ROW = 9_000
+SPIKE_VALUE = 999.0
+
+
+def _root() -> Path:
+    shutil.rmtree(SCRATCH, ignore_errors=True)
+    SCRATCH.mkdir(parents=True, exist_ok=True)
+    return SCRATCH
+
+
+def _packet(index: int) -> SdkPacket:
+    frame = np.zeros((4, 4, 3), np.uint8)
+    fx = SPIKE_VALUE if index == SPIKE_ROW else 0.5
+    return SdkPacket(
+        raw_frame=frame,
+        processed_frame=frame,
+        timestamp=1000.0 + index / 30.0,
+        mono_t=index / 30.0,
+        fps=30.0,
+        process_ms=1.0,
+        normal_force=1.0,
+        shear_x=fx,
+        shear_y=-0.5,
+        normal_force_raw=10.0 + (index % 7) * 0.01,
+        shear_x_raw=fx,
+        shear_y_raw=-0.5,
+        force_unit="N",
+        force_calibrated=True,
+        frame_id=index,
+    )
+
+
+def _recording() -> Path:
+    """A recording of ``ROWS`` rows with no drops: the writer is allowed to keep up."""
+    import time
+
+    root = _root()
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    directory = recorder.start(root=root, session_id=1, metadata={})
+    for index in range(ROWS):
+        recorder.on_packet(_packet(index))
+        if index % 256 == 0:
+            # Feeding 12000 rows in a tight loop would (correctly) overflow the
+            # ordered stream and be recorded as gaps; this test is about history
+            # browsing, so let the writer drain.
+            deadline = time.monotonic() + 5.0
+            while recorder._queue.qsize() > 512 and time.monotonic() < deadline:
+                time.sleep(0.001)
+    report = recorder.stop()
+    assert report.stats.dropped_data == 0, report.problems
+    assert report.stats.frames == ROWS
+    return Path(directory)
+
+
+def test_the_index_reads_only_the_interval_it_is_asked_for():
+    directory = _recording()
+    index = RecordingIndex(directory)
+    index.build()
+    assert index.rows == ROWS
+    assert index.duration_s() == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
+
+    # The whole range keeps every row, including the very first one.
+    whole = index.read_rows(0, ROWS)
+    assert len(whole) == ROWS
+    assert whole.frame_ids[0] == 0
+    assert whole.frame_ids[-1] == ROWS - 1
+
+    start, stop = index.row_range(300.0, 300.5)
+    # One row before the interval is included on purpose: a cursor exactly on a
+    # sample still wants the row it follows.
+    assert 15 <= stop - start <= 20, (start, stop)
+    window = index.read_rows(start, stop)
+    assert 15 <= len(window) <= 20
+    assert window.times[0] >= 300.0 - 1 / 30.0
+    assert window.times[-1] <= 300.5
+
+
+def test_a_large_recording_is_loaded_as_a_marked_decimated_view():
+    directory = _recording()
+    reader = RecordingReader(directory, max_rows=500)
+    info = reader.info()
+    assert info.decimated, "a 12000-row recording was loaded in full"
+    assert info.total_rows == ROWS
+    assert len(info.times) <= 500
+    # The duration comes from the frame index, never from the decimated view.
+    assert info.duration_s == pytest.approx((ROWS - 1) / 30.0, abs=0.05)
+    assert len(info.frames) == ROWS
+
+
+def test_a_small_recording_is_not_decimated():
+    directory = _recording()
+    reader = RecordingReader(directory, max_rows=50_000)
+    info = reader.info()
+    assert not info.decimated
+    assert len(info.times) == ROWS
+    assert info.total_rows == ROWS
+
+
+def test_an_interval_read_sees_every_row_including_a_single_peak():
+    directory = _recording()
+    reader = RecordingReader(directory, max_rows=500)
+    spike_t = SPIKE_ROW / 30.0
+    window = reader.window(spike_t - 0.05, spike_t + 0.05)
+    assert window.times.size >= 3
+    assert float(np.nanmax(window.value_for_unit("N")[:, 1])) == pytest.approx(SPIKE_VALUE)
+    # ...and it is the *exact* interval, not the decimated whole-session view.
+    assert not window.decimated
+    assert window.times[0] >= spike_t - 0.1
+
+
+def test_a_decimated_window_still_keeps_the_peak():
+    directory = _recording()
+    reader = RecordingReader(directory, max_rows=500)
+    window = reader.window(0.0, (ROWS - 1) / 30.0, max_points=300, unit="N")
+    assert window.decimated
+    assert len(window) <= 600
+    assert float(np.nanmax(window.value_for_unit("N")[:, 1])) == pytest.approx(SPIKE_VALUE)
+
+
+def test_the_envelope_covers_the_whole_session_in_few_points():
+    directory = _recording()
+    reader = RecordingReader(directory, max_rows=500)
+    overview = reader.overview(unit="N", buckets=200)
+    assert 0 < len(overview) <= 200
+    assert not np.isnan(overview.highs[:, 1]).all()
+    # The spike is inside a bucket the strip would otherwise skip.
+    assert float(np.nanmax(overview.highs[:, 1])) == pytest.approx(SPIKE_VALUE)
+    assert float(overview.counts.sum()) > 0
+    # Cached: asking again does not re-stream the file.
+    assert reader.overview(unit="N", buckets=200) is overview
+
+
+def test_the_two_sources_answer_the_same_questions():
+    directory = _recording()
+    reader = RecordingReader(directory, max_rows=ROWS + 1)
+    recording = RecordingCurveSource(reader)
+
+    store = SessionStore()
+    for index in range(ROWS):
+        sample = SessionSample(
+            index,
+            index / 30.0,
+            fn_raw=10.0 + (index % 7) * 0.01,
+            fx_raw=SPIKE_VALUE if index == SPIKE_ROW else 0.5,
+            fy_raw=-0.5,
+            fn_n=1.0,
+            fx_n=SPIKE_VALUE if index == SPIKE_ROW else 0.5,
+            fy_n=-0.5,
+        )
+        store.append(sample)
+    memory = StoreCurveSource(store)
+
+    for source in (recording, memory):
+        first, last = source.extent()
+        assert first == pytest.approx(0.0, abs=1e-6)
+        assert last == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
+        assert source.duration_s() == pytest.approx((ROWS - 1) / 30.0, abs=1e-6)
+    assert recording.kind == "disk" and memory.kind == "memory"
+
+    # The same interval, read from disk and from memory, agrees on the values.
+    # (The disk read also includes the row just before the interval.)
+    disk = recording.window(200.0, 210.0, unit="N")
+    cached = memory.window(200.0, 210.0, unit="N")
+    inside = (disk.times >= 200.0) & (disk.times <= 210.0)
+    assert inside.sum() == len(cached) > 0
+    assert np.allclose(disk.values[inside], cached.values, equal_nan=True)
+    # The CSV keeps six decimals; the in-memory store keeps the raw float.
+    assert np.allclose(disk.times[inside], cached.times, atol=1e-6)
+
+    # ...and so do the statistics of that interval (the disk read may include the
+    # one row that precedes it).
+    a = recording.statistics(200.0, 210.0, unit="N")
+    b = memory.statistics(200.0, 210.0, unit="N")
+    assert abs(a.samples - b.samples) <= 1
+    assert a.peaks[1] == pytest.approx(b.peaks[1])
+    assert a.means[1] == pytest.approx(b.means[1], abs=1e-6)
+
+
+def test_the_whole_session_can_be_read_without_holding_it_all():
+    """The point of the index: peak value with a bounded number of rows held."""
+    directory = _recording()
+    reader = RecordingReader(directory, max_rows=500)
+    overview = reader.overview(unit="N", buckets=100)
+    # What a whole-session strip needs is one band per bucket, not 12000 rows.
+    assert len(overview) <= 100
+    assert len(reader.info().times) <= 500
+    assert reader.info().total_rows == ROWS
+    # A single sample can still be read exactly, on demand.
+    spike_t = SPIKE_ROW / 30.0
+    exact = reader.window(spike_t, spike_t + 1 / 30.0)
+    assert 1 <= len(exact) <= 3, len(exact)
+    assert float(np.nanmax(exact.newtons[:, 1])) == pytest.approx(SPIKE_VALUE)

shared/tests/test_module_boundaries.py

@@ -0,0 +1,105 @@
+"""The module boundaries the responsibility split is supposed to create.
+
+A pure move is only worth anything if the dependency directions actually change:
+the finger worker must not need the box worker to describe a packet, the session
+file format must not need the sampling state machine, and the models must not
+import either. These assertions are cheap and they fail loudly if someone
+re-introduces the cycle.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+SHARED = Path(__file__).resolve().parents[1] / "src"
+ACQUISITION = SHARED / "orisys_app_shared" / "acquisition"
+FORCE = SHARED / "orisys_app_shared" / "force"
+RECORD = SHARED / "orisys_app_shared" / "record"
+
+
+def _imports(path: Path) -> set[str]:
+    """Every module name imported by ``path`` (``from x import y`` counts as x)."""
+    tree = ast.parse(path.read_text(encoding="utf-8"))
+    found: set[str] = set()
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Import):
+            for alias in node.names:
+                found.add(alias.name)
+        elif isinstance(node, ast.ImportFrom):
+            if node.module:
+                found.add(node.module)
+            # ``from . import models`` also names the submodule.
+            for alias in node.names:
+                found.add(f"{node.module}.{alias.name}" if node.module else alias.name)
+    return found
+
+
+def _imports_any(path: Path, needles) -> bool:
+    found = _imports(path)
+    for needle in needles:
+        if any(entry == needle or entry.endswith(f".{needle}") for entry in found):
+            return True
+    return False
+
+
+def test_the_packet_model_does_not_import_a_worker():
+    assert not _imports_any(
+        ACQUISITION / "models.py", {"sdk_worker", "finger_worker"}
+    ), "acquisition.models depends on a worker"
+
+
+def test_the_finger_worker_takes_the_packet_from_models():
+    """It used to import the box worker just to describe a frame."""
+    source = (ACQUISITION / "finger_worker.py").read_text(encoding="utf-8")
+    tree = ast.parse(source)
+    from_sdk_worker: set[str] = set()
+    for node in ast.walk(tree):
+        if isinstance(node, ast.ImportFrom) and (node.module or "").endswith("sdk_worker"):
+            from_sdk_worker.update(alias.name for alias in node.names)
+    assert "SdkPacket" not in from_sdk_worker, "the finger worker still borrows the packet"
+    assert "FixedCameraMode" not in from_sdk_worker
+    assert "load_fixed_mode" not in from_sdk_worker
+    # ...and the helpers it genuinely shares still come from there.
+    assert "process_chain" in from_sdk_worker
+
+
+def test_the_calibration_models_do_not_import_the_sampler_or_the_session_file():
+    for path in (FORCE / "models.py",):
+        assert not _imports_any(
+            path, {"calibration_recorder", "calibration_session", "session"}
+        ), f"{path.name} depends on a higher layer"
+
+
+def test_the_session_format_does_not_import_the_sampling_state_machine():
+    assert not _imports_any(
+        FORCE / "calibration_session.py", {"calibration_recorder"}
+    ), "the session file format still depends on the sampler"
+
+
+def test_the_moved_models_are_the_same_objects_everywhere():
+    from orisys_app_shared.acquisition import finger_worker, models, sdk_worker
+    from orisys_app_shared.force import calibration_recorder, calibration_session, models as force_models
+
+    assert sdk_worker.SdkPacket is models.SdkPacket is finger_worker.SdkPacket
+    assert sdk_worker.FixedCameraMode is models.FixedCameraMode
+    assert calibration_recorder.CalibrationPoint is force_models.CalibrationPoint
+    assert calibration_recorder.AxisStats is force_models.AxisStats
+    assert calibration_session.SAMPLE_FIELDS is force_models.SAMPLE_FIELDS
+    # The recorder still offers the trace helpers it used to own.
+    assert callable(calibration_recorder.trace_from)
+    assert callable(calibration_recorder.trace_dt)
+
+
+def test_the_recording_pieces_are_separate_modules():
+    from orisys_app_shared.record import depth, experiment
+    from orisys_app_shared.record.depth_backends import BoxReplay, FingerReplay
+    from orisys_app_shared.record.video_writer import VideoSegments
+
+    # The depth replay no longer defines its backends inline...
+    assert not hasattr(depth, "BoxReplay")
+    assert not hasattr(depth, "_BoxReplay")
+    # ...and the writer delegates video encoding to its own component.
+    assert not hasattr(experiment, "_open_segment")
+    assert callable(VideoSegments.write) and callable(VideoSegments.close)
+    assert BoxReplay.__name__ == "BoxReplay" and FingerReplay.__name__ == "FingerReplay"

shared/tests/test_playback.py

@@ -137,6 +137,78 @@ def test_reader_window_and_streaming_agree(recording):
     assert streamed == 61, "streaming must see every row"
 
 
+def test_an_empty_recording_directory_is_readable():
+    """A folder with no CSVs must open (empty), not raise an unpack error."""
+    shutil.rmtree(SCRATCH, ignore_errors=True)
+    SCRATCH.mkdir(parents=True, exist_ok=True)
+    try:
+        reader = RecordingReader(SCRATCH)
+        info = reader.info()
+        assert info.frames == []
+        assert info.times.size == 0
+        assert info.duration_s == 0.0
+        assert reader.frame_at(1.0) is None
+        assert reader.replay_warnings(sdk_version="0.5.1")
+    finally:
+        shutil.rmtree(SCRATCH, ignore_errors=True)
+
+
+def test_a_new_recording_is_reported_as_time_ordered(recording):
+    info = RecordingReader(recording).info()
+    assert info.frames_out_of_order, "the writer no longer produces ordered rows"
+    assert info.forces_out_of_order
+    codes = RecordingReader(recording).replay_warnings(sdk_version="0.5.1")
+    assert "out_of_order" not in codes
+
+
+def test_an_old_out_of_order_recording_is_re_sorted_and_reported():
+    """Rows written by the pre-fix writer are not in time order.
+
+    ``frame_at`` binary-searches these times, so an unsorted file would answer
+    with the wrong frame; the reader sorts and says so instead.
+    """
+    shutil.rmtree(SCRATCH, ignore_errors=True)
+    SCRATCH.mkdir(parents=True, exist_ok=True)
+    (SCRATCH / "forces.csv").write_text(
+        "session_id,frame_id,t_rel_s,FN_raw,FX_raw,FY_raw,FN_N,FX_N,FY_N,valid,calibration_id\n"
+        "1,0,0.0,1.0,0.1,0.1,2.0,0.1,0.1,1,h\n"
+        "1,2,0.066,3.0,0.1,0.1,6.0,0.1,0.1,1,h\n"  # written before frame 1
+        "1,1,0.033,2.0,0.1,0.1,4.0,0.1,0.1,1,h\n",
+        encoding="utf-8",
+    )
+    (SCRATCH / "frames.csv").write_text(
+        "frame_id,t_rel_s,acquired_at,segment,segment_frame,in_video,dropped\n"
+        "0,0.0,2026-01-01T00:00:00.000Z,1,1,1,0\n"
+        "2,0.066,2026-01-01T00:00:00.066Z,1,3,1,0\n"
+        "1,0.033,2026-01-01T00:00:00.033Z,1,2,1,0\n",
+        encoding="utf-8",
+    )
+    (SCRATCH / "session.json").write_text(
+        json.dumps(
+            {
+                "format_version": "1.1",
+                "device": {"sensor_type": "box", "config_ref": "/cfg/sensor.json"},
+                "versions": {"sdk": "0.5.1"},
+                "depth": {"stored": False},
+            }
+        ),
+        encoding="utf-8",
+    )
+    try:
+        reader = RecordingReader(SCRATCH)
+        info = reader.info()
+        assert not info.frames_out_of_order
+        assert not info.forces_out_of_order
+        assert [frame.frame_id for frame in info.frames] == [0, 1, 2]
+        assert info.times.tolist() == pytest.approx([0.0, 0.033, 0.066])
+        # The binary search is only valid because the rows were sorted.
+        assert reader.frame_at(0.05).frame_id == 1
+        assert reader.frame_at(0.07).frame_id == 2
+        assert "out_of_order" in reader.replay_warnings(sdk_version="0.5.1")
+    finally:
+        shutil.rmtree(SCRATCH, ignore_errors=True)
+
+
 def test_depth_policy_says_recompute_and_warns_about_drift(recording):
     reader = RecordingReader(recording)
 

shared/tests/test_record_lifecycle.py

@@ -10,11 +10,9 @@ from __future__ import annotations
 
 import csv
 import json
-import queue
 import shutil
 import threading
 import time
-from collections import deque
 from pathlib import Path
 from unittest.mock import patch
 
@@ -23,7 +21,11 @@ import pytest
 
 from orisys_app_shared.acquisition.sdk_worker import SdkPacket
 from orisys_app_shared.record import ExperimentRecorder, RecordConfig
-from orisys_app_shared.record.experiment import STATE_FAILED, STATE_STOPPED, _SessionWriter
+from orisys_app_shared.record.experiment import (
+    STATE_FAILED,
+    STATE_STOPPED,
+    _SessionWriter,
+)
 
 SCRATCH = Path(__file__).resolve().parent / "_scratch"
 
@@ -189,23 +191,30 @@ def test_a_full_queue_keeps_the_rows_and_sheds_the_video():
 
 
 def test_a_lost_row_is_recorded_as_an_index_gap():
-    """A hole in the data must be visible, not silently bridged in replay."""
+    """A hole in the data must be visible, not silently bridged in replay.
+
+    The row is only given up when the ordered stream itself is full; the gap
+    marker then carries the stream position, so it lands in the index where the
+    hole actually is instead of after newer frames.
+    """
     original = _SessionWriter.write_sample
     calls = {"count": 0}
 
     def slow(self, *args):
         calls["count"] += 1
         if calls["count"] <= 2:
-            # Stall just long enough for the queue and the tiny backlog to fill.
+            # Stall just long enough for the stream to pass the (tiny) cap.
             time.sleep(0.3)
         return original(self, *args)
 
     recorder = ExperimentRecorder(RecordConfig(video=False, max_queue=16))
     with patch.object(_SessionWriter, "write_sample", slow):
         directory = recorder.start(root=_root(), session_id=1, metadata={})
-        recorder._row_backlog = deque(recorder._row_backlog, maxlen=2)
-        for index in range(40):
-            recorder.on_packet(_packet(index))
+        with patch(
+            "orisys_app_shared.record.experiment.ROW_BACKLOG", 2
+        ):
+            for index in range(40):
+                recorder.on_packet(_packet(index))
         report = recorder.stop()
 
     with (Path(directory) / "frames.csv").open(encoding="utf-8", newline="") as handle:
@@ -218,6 +227,57 @@ def test_a_lost_row_is_recorded_as_an_index_gap():
     assert report.stats.dropped_gaps > 0, "the loss was not marked in the index"
 
 
+def test_the_frame_index_stays_in_time_order_when_rows_are_lost():
+    """The old writer drained its backlog before the queue, so the index jumped."""
+    original = _SessionWriter.write_sample
+    calls = {"count": 0}
+
+    def slow(self, *args):
+        calls["count"] += 1
+        if calls["count"] <= 3:
+            time.sleep(0.2)
+        return original(self, *args)
+
+    recorder = ExperimentRecorder(RecordConfig(video=True, max_queue=4))
+    with patch.object(_SessionWriter, "write_sample", slow):
+        directory = recorder.start(root=_root(), session_id=1, metadata={})
+        with patch("orisys_app_shared.record.experiment.ROW_BACKLOG", 3):
+            for index in range(60):
+                recorder.on_packet(_packet(index))
+        recorder.stop()
+
+    with (Path(directory) / "frames.csv").open(encoding="utf-8", newline="") as handle:
+        frame_rows = list(csv.DictReader(handle))
+    times = [float(row["t_rel_s"]) for row in frame_rows]
+    assert times == sorted(times), "frames.csv is not in time order"
+    ids = [int(row["frame_id"]) for row in frame_rows]
+    assert ids == sorted(ids), "forces.csv/frames.csv disagree on order"
+    with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
+        force_rows = list(csv.DictReader(handle))
+    force_times = [float(row["t_rel_s"]) for row in force_rows]
+    assert force_times == sorted(force_times), "forces.csv is not in time order"
+
+
+def test_the_video_byte_budget_is_released_when_a_frame_is_shed():
+    """A shed frame must not leak its reservation for the rest of the session."""
+    recorder = ExperimentRecorder(
+        RecordConfig(video=True, max_queue=16, max_queue_bytes=4096)
+    )
+    directory = recorder.start(root=_root(), session_id=1, metadata={})
+    # Frames of 8x8x3 = 192 bytes, so the budget holds ~21 of them; without the
+    # release the counter would climb past the budget and stay there.
+    for index in range(200):
+        recorder.on_packet(_packet(index))
+    report = recorder.stop()
+    assert report.stats.frames == 200, "rows were lost to the video budget"
+    assert abs(recorder._queued_bytes) < 4096, (
+        f"the byte budget leaked: {recorder._queued_bytes} bytes still reserved"
+    )
+    with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
+        rows = list(csv.DictReader(handle))
+    assert len(rows) == 200
+
+
 def test_session_json_names_the_pre_roll_when_one_was_written():
     frames = [np.full((8, 8, 3), index, np.uint8) for index in range(3)]
     recorder = ExperimentRecorder(RecordConfig(video=True, fps=5.0))
@@ -241,6 +301,80 @@ def test_a_data_only_recording_says_why_it_has_no_pre_roll():
     assert any("pre-roll" in problem for problem in report.problems)
 
 
+def test_a_stalled_encoder_leaves_a_readable_recording():
+    """Order, duration, positioning and the byte counter after a stall.
+
+    Stall the writer, push past the ordered stream's cap, let it recover, then
+    push again and read the directory back the way the replay does. The frames
+    acquired *after* the recovery must all be there: an overflow is a temporary
+    loss, not the end of the recording.
+    """
+    from orisys_app_shared.record import RecordingReader
+
+    original = _SessionWriter.write_sample
+    started = threading.Event()
+    release = threading.Event()
+    calls = {"count": 0}
+
+    def slow(self, *args):
+        calls["count"] += 1
+        if calls["count"] == 2:
+            started.set()
+            release.wait(10.0)
+        return original(self, *args)
+
+    def drained(recorder, timeout: float = 10.0) -> bool:
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if recorder._queue.qsize() == 0:
+                return True
+            time.sleep(0.01)
+        return False
+
+    recorder = ExperimentRecorder(RecordConfig(video=True, fps=30.0, max_queue=4))
+    with patch.object(_SessionWriter, "write_sample", slow):
+        directory = recorder.start(root=_root(), session_id=1, metadata={})
+        with patch("orisys_app_shared.record.experiment.ROW_BACKLOG", 3):
+            for index in range(40):
+                recorder.on_packet(_packet(index))
+            assert started.wait(10.0), "the stall never happened"
+            release.set()
+            assert drained(recorder), "the writer never caught up"
+            before = recorder.stats().frames
+            for index in range(40, 100):
+                recorder.on_packet(_packet(index))
+                time.sleep(0.005)
+            assert drained(recorder), "the recovered writer never caught up"
+        report = recorder.stop()
+
+    assert not report.stats.failed, report.problems
+    assert recorder.stats().frames > before, "recording did not continue after the stall"
+
+    reader = RecordingReader(directory)
+    info = reader.info()
+    times = info.times.tolist()
+    ids = [int(value) for value in info.frame_ids]
+    assert times == sorted(times), "forces.csv is not in time order"
+    assert ids == sorted(ids)
+    frame_times = [frame.t_rel_s for frame in info.frames]
+    assert frame_times == sorted(frame_times), "frames.csv is not in time order"
+
+    # Everything acquired after the recovery is present.
+    assert set(range(40, 100)).issubset(set(ids)), (
+        f"frames lost after the encoder recovered: {sorted(set(range(40, 100)) - set(ids))}"
+    )
+    assert report.stats.duration_s == pytest.approx(99 / 30.0, abs=0.05)
+    assert info.duration_s == pytest.approx(99 / 30.0, abs=0.05)
+
+    # Positioning: the frame at 2 s is the frame acquired at 2 s.
+    frame = reader.frame_at(2.0)
+    assert frame is not None and frame.frame_id == 60, frame
+    assert frame.t_rel_s == pytest.approx(2.0, abs=0.05)
+
+    # The byte budget is back to zero once every frame has been consumed.
+    assert recorder._queued_bytes == 0, recorder._queued_bytes
+
+
 def test_stopping_twice_returns_the_same_report():
     recorder = ExperimentRecorder(RecordConfig(video=False))
     recorder.start(root=_root(), session_id=1, metadata={})
@@ -249,3 +383,146 @@ def test_stopping_twice_returns_the_same_report():
     second = recorder.stop()
     assert first.directory == second.directory
     assert second.ok
+
+
+# ------------------------------------------------------------ failure handling
+
+
+def test_a_directory_that_cannot_be_created_leaves_a_retryable_state():
+    """A failed ``mkdir`` must not strand the recorder in ``starting``."""
+    root = _root()
+    blocker = root / "not-a-directory"
+    blocker.write_text("", encoding="utf-8")
+
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    with pytest.raises(Exception):
+        recorder.start(root=blocker, session_id=1, metadata={})
+    assert recorder.state == STATE_FAILED
+    assert not recorder.is_recording
+    assert recorder.directory is None
+    assert recorder.stats().last_error
+
+    # Retryable: the operator fixes the path and presses record again.
+    directory = recorder.start(root=root, session_id=2, metadata={})
+    recorder.on_packet(_packet(0))
+    report = recorder.stop()
+    assert report.ok, report.problems
+    assert Path(directory).is_dir()
+
+
+def test_a_stop_during_initialization_never_resumes_recording():
+    """``stop`` that lands while the files open must not be undone afterwards."""
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    original = ExperimentRecorder._write_session_json
+    writing = threading.Event()
+    release = threading.Event()
+    calls = {"count": 0}
+
+    def gated(self, *, final: bool) -> None:
+        calls["count"] += 1
+        if calls["count"] == 1:
+            # This is the writer thread's first metadata write.
+            writing.set()
+            release.wait(5.0)
+        return original(self, final=final)
+
+    with patch.object(ExperimentRecorder, "_write_session_json", gated):
+        directory = recorder.start(root=_root(), session_id=1, metadata={})
+        recorder.on_packet(_packet(0))
+        assert writing.wait(5.0)
+        report = recorder.stop(wait=False)
+        release.set()
+        thread = recorder._thread
+        if thread is not None:
+            thread.join(5.0)
+
+    # The row accepted before the stop is still written...
+    with (Path(directory) / "forces.csv").open(encoding="utf-8", newline="") as handle:
+        rows = list(csv.DictReader(handle))
+    assert len(rows) == 1, "rows accepted before the stop were dropped"
+    # ...but the recording never went back to "recording", and it is finished.
+    assert not recorder.is_recording
+    assert recorder.state in (STATE_STOPPED, STATE_FAILED)
+    assert not report.completed or report.directory is not None
+
+
+def test_stop_without_waiting_does_not_claim_completion():
+    started = threading.Event()
+    release = threading.Event()
+    original = _SessionWriter.write_sample
+
+    def slow(self, *args):
+        started.set()
+        release.wait(10.0)
+        return original(self, *args)
+
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    with patch.object(_SessionWriter, "write_sample", slow):
+        recorder.start(root=_root(), session_id=1, metadata={})
+        recorder.on_packet(_packet(0))
+        assert started.wait(10.0)
+        report = recorder.stop(wait=False)
+        try:
+            assert not report.completed
+            assert not report.ok
+            assert recorder.report is None, (
+                "an unfinished stop was cached as the terminal report"
+            )
+        finally:
+            release.set()
+            recorder._thread.join(5.0)
+    assert recorder.report is not None and recorder.report.completed
+
+
+def test_a_failing_flush_reaches_the_state_machine_at_once():
+    """A flush failure used to live only in the writer's private counter."""
+    recorder = ExperimentRecorder(RecordConfig(video=False, flush_interval_s=0.0))
+    directory = recorder.start(root=_root(), session_id=1, metadata={})
+
+    class _BrokenHandle:
+        def flush(self):
+            raise OSError("input/output error")
+
+        def close(self):
+            return None
+
+    writer = _SessionWriter(
+        directory=Path(directory),
+        config=recorder.config,
+        session_id=1,
+        calibration_id="",
+        on_problem=recorder._record_failure,
+    )
+    writer._forces = (_BrokenHandle(), None)
+    writer.maybe_flush()
+    assert recorder.stats().write_errors == 1, "the flush failure was not reported"
+    assert "flush failed" in recorder.stats().last_error
+
+    for _ in range(4):
+        writer.maybe_flush()
+    assert recorder.stats().failed, "repeated flush failures never failed the record"
+    assert not recorder.is_recording
+    recorder.stop()
+
+
+def test_metadata_write_failure_is_part_of_the_report():
+    """A session.json that cannot be written is part of what happened."""
+    real_write_text = Path.write_text
+
+    def failing_write_text(self, *args, **kwargs):
+        if self.name.endswith(".tmp"):
+            raise OSError("read-only filesystem")
+        return real_write_text(self, *args, **kwargs)
+
+    recorder = ExperimentRecorder(RecordConfig(video=False))
+    with patch.object(Path, "write_text", failing_write_text):
+        recorder.start(root=_root(), session_id=1, metadata={})
+        recorder.on_packet(_packet(0))
+        report = recorder.stop()
+
+    assert report.stats.write_errors >= 1
+    assert not report.ok, "a missing session.json was reported as a good recording"
+    assert any(
+        "read-only" in problem or "write error" in problem for problem in report.problems
+    ), report.problems
+    assert report.stats.failed