Browse a long recording from disk instead of loading it into memory

What
- record/index.py: RecordingIndex streams forces.csv once and keeps only the
  small things — one time, one validity flag, one frame id per row plus a byte
  offset every 4096 rows (~6.5 MB per 500k rows). row_range() bisects the times,
  read_rows() seeks to the nearest checkpoint and scans forward, and overview()
  folds every row into its bucket during the same pass to build a per-axis
  min/max envelope.
- read_rows(max_points, unit) decimates **while streaming**: each bucket keeps
  the extreme row of every axis in the requested unit, so a long interval never
  has to exist in memory in full and a shear spike survives a flat normal channel.
- RecordingReader: info() decimates above max_rows (default 200k) and marks
  itself decimated, with total_rows and a duration taken from the frame index
  rather than from the view; window()/window_all()/overview()/statistics() read
  the interval being asked about. list/window semantics unchanged for small
  recordings.
- force/source.py: StoreCurveSource (live memory) and RecordingCurveSource (disk
  interval + cached envelope) implement one interface — extent, window,
  envelope, statistics, duration_s — so ForcePlotsPanel asks a *source* instead
  of knowing where the data lives.
- force_plots: reads through the source (window, envelope, statistics, extent),
  which also replaces its last direct store accesses.
- playback_dialog: draws the whole-session view through window_all() in the unit
  being shown (bounded by CHART_MAX_POINTS), and says so when the view is
  decimated ("显示 N / 共 M 行") instead of letting the operator count points.
- experiment._run: a failure that lands while the files are opening can no longer
  be overwritten by the writer's own transition to "recording" (found by a flaky
  test; the state check now covers STATE_FAILED).

Why
The store releases old blocks above ~4.6 h and the replay loaded every CSV row
into Python lists before drawing anything, so neither the live browser nor the
replay could show an earlier interval of a long recording. The index makes the
disk the data source and memory a cache of what is on screen.

Verification
- shared/tests/test_history_source.py (new, 8 cases) over a 12 000-row recording:
  the index reports its rows and reads a bounded interval; a large recording is
  marked decimated with its true duration and total rows; a small one is not; an
  interval read sees the exact rows including a single peak; a decimated window
  keeps the peak; the envelope fits the whole session in ≤200 points, keeps the
  spike and is cached; the disk and memory sources agree on extent, values and
  statistics; the whole session can be read without holding it all. Shared
  suites: 146 tests, 0 failures (three consecutive runs).
- apps/studio/tools/ui_check.py: the replay section now checks the index row
  count, that an interval read spans only the interval, the envelope size and
  total_rows. 324 checks, 0 failures.
- apps/studio/tests/test_ui_smoke.py: 43 tests, 36 pass; the 7 failures are the
  unchanged sandbox/harness artifacts.
- D:/orisys_ws/_review_phase12/reproduce.py: all ten lines unchanged;
  compileall, locale and prefs parity clean.
- apps/studio/docs/record-channel.md §8 documents the index, the reader API and
  the unified source, and lists what is still hardware work.

Not in this change
- Maximum RSS / backlog / loss measurements at real resolution on the target ARM
  board (needs the hardware), and range CSV export.
This commit is contained in:
ykcheungab
2026-09-13 16:54:26 +08:00
parent 22cc59c402
commit 73592f085f
40 changed files with 6398 additions and 1070 deletions
+72
View File
@@ -137,6 +137,78 @@ def test_reader_window_and_streaming_agree(recording):
assert streamed == 61, "streaming must see every row"
def test_an_empty_recording_directory_is_readable():
"""A folder with no CSVs must open (empty), not raise an unpack error."""
shutil.rmtree(SCRATCH, ignore_errors=True)
SCRATCH.mkdir(parents=True, exist_ok=True)
try:
reader = RecordingReader(SCRATCH)
info = reader.info()
assert info.frames == []
assert info.times.size == 0
assert info.duration_s == 0.0
assert reader.frame_at(1.0) is None
assert reader.replay_warnings(sdk_version="0.5.1")
finally:
shutil.rmtree(SCRATCH, ignore_errors=True)
def test_a_new_recording_is_reported_as_time_ordered(recording):
info = RecordingReader(recording).info()
assert info.frames_out_of_order, "the writer no longer produces ordered rows"
assert info.forces_out_of_order
codes = RecordingReader(recording).replay_warnings(sdk_version="0.5.1")
assert "out_of_order" not in codes
def test_an_old_out_of_order_recording_is_re_sorted_and_reported():
"""Rows written by the pre-fix writer are not in time order.
``frame_at`` binary-searches these times, so an unsorted file would answer
with the wrong frame; the reader sorts and says so instead.
"""
shutil.rmtree(SCRATCH, ignore_errors=True)
SCRATCH.mkdir(parents=True, exist_ok=True)
(SCRATCH / "forces.csv").write_text(
"session_id,frame_id,t_rel_s,FN_raw,FX_raw,FY_raw,FN_N,FX_N,FY_N,valid,calibration_id\n"
"1,0,0.0,1.0,0.1,0.1,2.0,0.1,0.1,1,h\n"
"1,2,0.066,3.0,0.1,0.1,6.0,0.1,0.1,1,h\n" # written before frame 1
"1,1,0.033,2.0,0.1,0.1,4.0,0.1,0.1,1,h\n",
encoding="utf-8",
)
(SCRATCH / "frames.csv").write_text(
"frame_id,t_rel_s,acquired_at,segment,segment_frame,in_video,dropped\n"
"0,0.0,2026-01-01T00:00:00.000Z,1,1,1,0\n"
"2,0.066,2026-01-01T00:00:00.066Z,1,3,1,0\n"
"1,0.033,2026-01-01T00:00:00.033Z,1,2,1,0\n",
encoding="utf-8",
)
(SCRATCH / "session.json").write_text(
json.dumps(
{
"format_version": "1.1",
"device": {"sensor_type": "box", "config_ref": "/cfg/sensor.json"},
"versions": {"sdk": "0.5.1"},
"depth": {"stored": False},
}
),
encoding="utf-8",
)
try:
reader = RecordingReader(SCRATCH)
info = reader.info()
assert not info.frames_out_of_order
assert not info.forces_out_of_order
assert [frame.frame_id for frame in info.frames] == [0, 1, 2]
assert info.times.tolist() == pytest.approx([0.0, 0.033, 0.066])
# The binary search is only valid because the rows were sorted.
assert reader.frame_at(0.05).frame_id == 1
assert reader.frame_at(0.07).frame_id == 2
assert "out_of_order" in reader.replay_warnings(sdk_version="0.5.1")
finally:
shutil.rmtree(SCRATCH, ignore_errors=True)
def test_depth_policy_says_recompute_and_warns_about_drift(recording):
reader = RecordingReader(recording)