Browse a long recording from disk instead of loading it into memory

What
- record/index.py: RecordingIndex streams forces.csv once and keeps only the
  small things — one time, one validity flag, one frame id per row plus a byte
  offset every 4096 rows (~6.5 MB per 500k rows). row_range() bisects the times,
  read_rows() seeks to the nearest checkpoint and scans forward, and overview()
  folds every row into its bucket during the same pass to build a per-axis
  min/max envelope.
- read_rows(max_points, unit) decimates **while streaming**: each bucket keeps
  the extreme row of every axis in the requested unit, so a long interval never
  has to exist in memory in full and a shear spike survives a flat normal channel.
- RecordingReader: info() decimates above max_rows (default 200k) and marks
  itself decimated, with total_rows and a duration taken from the frame index
  rather than from the view; window()/window_all()/overview()/statistics() read
  the interval being asked about. list/window semantics unchanged for small
  recordings.
- force/source.py: StoreCurveSource (live memory) and RecordingCurveSource (disk
  interval + cached envelope) implement one interface — extent, window,
  envelope, statistics, duration_s — so ForcePlotsPanel asks a *source* instead
  of knowing where the data lives.
- force_plots: reads through the source (window, envelope, statistics, extent),
  which also replaces its last direct store accesses.
- playback_dialog: draws the whole-session view through window_all() in the unit
  being shown (bounded by CHART_MAX_POINTS), and says so when the view is
  decimated ("显示 N / 共 M 行") instead of letting the operator count points.
- experiment._run: a failure that lands while the files are opening can no longer
  be overwritten by the writer's own transition to "recording" (found by a flaky
  test; the state check now covers STATE_FAILED).

Why
The store releases old blocks above ~4.6 h and the replay loaded every CSV row
into Python lists before drawing anything, so neither the live browser nor the
replay could show an earlier interval of a long recording. The index makes the
disk the data source and memory a cache of what is on screen.

Verification
- shared/tests/test_history_source.py (new, 8 cases) over a 12 000-row recording:
  the index reports its rows and reads a bounded interval; a large recording is
  marked decimated with its true duration and total rows; a small one is not; an
  interval read sees the exact rows including a single peak; a decimated window
  keeps the peak; the envelope fits the whole session in ≤200 points, keeps the
  spike and is cached; the disk and memory sources agree on extent, values and
  statistics; the whole session can be read without holding it all. Shared
  suites: 146 tests, 0 failures (three consecutive runs).
- apps/studio/tools/ui_check.py: the replay section now checks the index row
  count, that an interval read spans only the interval, the envelope size and
  total_rows. 324 checks, 0 failures.
- apps/studio/tests/test_ui_smoke.py: 43 tests, 36 pass; the 7 failures are the
  unchanged sandbox/harness artifacts.
- D:/orisys_ws/_review_phase12/reproduce.py: all ten lines unchanged;
  compileall, locale and prefs parity clean.
- apps/studio/docs/record-channel.md §8 documents the index, the reader API and
  the unified source, and lists what is still hardware work.

Not in this change
- Maximum RSS / backlog / loss measurements at real resolution on the target ARM
  board (needs the hardware), and range CSV export.
This commit is contained in:
ykcheungab
2026-09-13 16:54:26 +08:00
parent 22cc59c402
commit 73592f085f
40 changed files with 6398 additions and 1070 deletions
+105
View File
@@ -0,0 +1,105 @@
"""The module boundaries the responsibility split is supposed to create.
A pure move is only worth anything if the dependency directions actually change:
the finger worker must not need the box worker to describe a packet, the session
file format must not need the sampling state machine, and the models must not
import either. These assertions are cheap and they fail loudly if someone
re-introduces the cycle.
"""
from __future__ import annotations
import ast
from pathlib import Path
SHARED = Path(__file__).resolve().parents[1] / "src"
ACQUISITION = SHARED / "orisys_app_shared" / "acquisition"
FORCE = SHARED / "orisys_app_shared" / "force"
RECORD = SHARED / "orisys_app_shared" / "record"
def _imports(path: Path) -> set[str]:
"""Every module name imported by ``path`` (``from x import y`` counts as x)."""
tree = ast.parse(path.read_text(encoding="utf-8"))
found: set[str] = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for alias in node.names:
found.add(alias.name)
elif isinstance(node, ast.ImportFrom):
if node.module:
found.add(node.module)
# ``from . import models`` also names the submodule.
for alias in node.names:
found.add(f"{node.module}.{alias.name}" if node.module else alias.name)
return found
def _imports_any(path: Path, needles) -> bool:
found = _imports(path)
for needle in needles:
if any(entry == needle or entry.endswith(f".{needle}") for entry in found):
return True
return False
def test_the_packet_model_does_not_import_a_worker():
assert not _imports_any(
ACQUISITION / "models.py", {"sdk_worker", "finger_worker"}
), "acquisition.models depends on a worker"
def test_the_finger_worker_takes_the_packet_from_models():
"""It used to import the box worker just to describe a frame."""
source = (ACQUISITION / "finger_worker.py").read_text(encoding="utf-8")
tree = ast.parse(source)
from_sdk_worker: set[str] = set()
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and (node.module or "").endswith("sdk_worker"):
from_sdk_worker.update(alias.name for alias in node.names)
assert "SdkPacket" not in from_sdk_worker, "the finger worker still borrows the packet"
assert "FixedCameraMode" not in from_sdk_worker
assert "load_fixed_mode" not in from_sdk_worker
# ...and the helpers it genuinely shares still come from there.
assert "process_chain" in from_sdk_worker
def test_the_calibration_models_do_not_import_the_sampler_or_the_session_file():
for path in (FORCE / "models.py",):
assert not _imports_any(
path, {"calibration_recorder", "calibration_session", "session"}
), f"{path.name} depends on a higher layer"
def test_the_session_format_does_not_import_the_sampling_state_machine():
assert not _imports_any(
FORCE / "calibration_session.py", {"calibration_recorder"}
), "the session file format still depends on the sampler"
def test_the_moved_models_are_the_same_objects_everywhere():
from orisys_app_shared.acquisition import finger_worker, models, sdk_worker
from orisys_app_shared.force import calibration_recorder, calibration_session, models as force_models
assert sdk_worker.SdkPacket is models.SdkPacket is finger_worker.SdkPacket
assert sdk_worker.FixedCameraMode is models.FixedCameraMode
assert calibration_recorder.CalibrationPoint is force_models.CalibrationPoint
assert calibration_recorder.AxisStats is force_models.AxisStats
assert calibration_session.SAMPLE_FIELDS is force_models.SAMPLE_FIELDS
# The recorder still offers the trace helpers it used to own.
assert callable(calibration_recorder.trace_from)
assert callable(calibration_recorder.trace_dt)
def test_the_recording_pieces_are_separate_modules():
from orisys_app_shared.record import depth, experiment
from orisys_app_shared.record.depth_backends import BoxReplay, FingerReplay
from orisys_app_shared.record.video_writer import VideoSegments
# The depth replay no longer defines its backends inline...
assert not hasattr(depth, "BoxReplay")
assert not hasattr(depth, "_BoxReplay")
# ...and the writer delegates video encoding to its own component.
assert not hasattr(experiment, "_open_segment")
assert callable(VideoSegments.write) and callable(VideoSegments.close)
assert BoxReplay.__name__ == "BoxReplay" and FingerReplay.__name__ == "FingerReplay"