Browse a long recording from disk instead of loading it into memory
What
- record/index.py: RecordingIndex streams forces.csv once and keeps only the
small things — one time, one validity flag, one frame id per row plus a byte
offset every 4096 rows (~6.5 MB per 500k rows). row_range() bisects the times,
read_rows() seeks to the nearest checkpoint and scans forward, and overview()
folds every row into its bucket during the same pass to build a per-axis
min/max envelope.
- read_rows(max_points, unit) decimates **while streaming**: each bucket keeps
the extreme row of every axis in the requested unit, so a long interval never
has to exist in memory in full and a shear spike survives a flat normal channel.
- RecordingReader: info() decimates above max_rows (default 200k) and marks
itself decimated, with total_rows and a duration taken from the frame index
rather than from the view; window()/window_all()/overview()/statistics() read
the interval being asked about. list/window semantics unchanged for small
recordings.
- force/source.py: StoreCurveSource (live memory) and RecordingCurveSource (disk
interval + cached envelope) implement one interface — extent, window,
envelope, statistics, duration_s — so ForcePlotsPanel asks a *source* instead
of knowing where the data lives.
- force_plots: reads through the source (window, envelope, statistics, extent),
which also replaces its last direct store accesses.
- playback_dialog: draws the whole-session view through window_all() in the unit
being shown (bounded by CHART_MAX_POINTS), and says so when the view is
decimated ("显示 N / 共 M 行") instead of letting the operator count points.
- experiment._run: a failure that lands while the files are opening can no longer
be overwritten by the writer's own transition to "recording" (found by a flaky
test; the state check now covers STATE_FAILED).
Why
The store releases old blocks above ~4.6 h and the replay loaded every CSV row
into Python lists before drawing anything, so neither the live browser nor the
replay could show an earlier interval of a long recording. The index makes the
disk the data source and memory a cache of what is on screen.
Verification
- shared/tests/test_history_source.py (new, 8 cases) over a 12 000-row recording:
the index reports its rows and reads a bounded interval; a large recording is
marked decimated with its true duration and total rows; a small one is not; an
interval read sees the exact rows including a single peak; a decimated window
keeps the peak; the envelope fits the whole session in ≤200 points, keeps the
spike and is cached; the disk and memory sources agree on extent, values and
statistics; the whole session can be read without holding it all. Shared
suites: 146 tests, 0 failures (three consecutive runs).
- apps/studio/tools/ui_check.py: the replay section now checks the index row
count, that an interval read spans only the interval, the envelope size and
total_rows. 324 checks, 0 failures.
- apps/studio/tests/test_ui_smoke.py: 43 tests, 36 pass; the 7 failures are the
unchanged sandbox/harness artifacts.
- D:/orisys_ws/_review_phase12/reproduce.py: all ten lines unchanged;
compileall, locale and prefs parity clean.
- apps/studio/docs/record-channel.md §8 documents the index, the reader API and
the unified source, and lists what is still hardware work.
Not in this change
- Maximum RSS / backlog / loss measurements at real resolution on the target ARM
board (needs the hardware), and range CSV export.
This commit is contained in:
@@ -293,9 +293,44 @@ outputs/recordings/<UTC 时间戳>_session<NNNN>/
|
||||
|
||||
---
|
||||
|
||||
## 8. 尚未完成(第三阶段及以后)
|
||||
## 8. 长历史浏览:按需读盘而不是整段载入
|
||||
|
||||
`record/index.py` 的 `RecordingIndex` 是回放的数据源索引:
|
||||
|
||||
- 流式扫一遍 `forces.csv`,只把**小东西**留在内存:每行一个时间(float64)、一个
|
||||
valid 标志、一个帧号,以及每 `CHECKPOINT_ROWS`(4096)行的字节偏移。500k 行约
|
||||
6.5 MB(`SessionStore` 的默认上限正好对应 30 Hz 下约 4.6 小时)。
|
||||
- `row_range(t0, t1)` 用二分定位行区间,`read_rows(start, stop)` 先 `seek` 到最近的
|
||||
检查点再向前扫描——读一个 10 秒区间就是 10 秒的解析量,不是整段。
|
||||
区间会额外带上 `t0` 之前的那一行,绘制时才能画出跨过左边界的线段。
|
||||
- `overview(buckets)` 在**同一次流式扫描**里把每行折进所属桶,得到每桶每轴的
|
||||
min/max/count。全程色带因此不会漏掉某桶里的尖峰(按行抽样会漏)。
|
||||
|
||||
`ReadingReader` 相应变化:
|
||||
|
||||
| 方法 | 作用 |
|
||||
|---|---|
|
||||
| `info()` | 行数超过 `max_rows`(默认 200k)时给出**降采样视图**,并置 `decimated=True`;`duration_s` 始终取自帧索引,不取自视图 |
|
||||
| `window(t0, t1, max_points=…)` | 区间精确读取(可再降采样),尖峰按轴保留 |
|
||||
| `overview(unit, buckets)` | 全程包络,按 (unit, buckets) 缓存 |
|
||||
| `statistics(t0, t1, unit)` | 只算该区间的统计量 |
|
||||
|
||||
`force/source.py` 把两种来源统一成同一接口:`StoreCurveSource`(实时内存块)与
|
||||
`RecordingCurveSource`(磁盘区间 + 缓存包络),都提供
|
||||
`extent() / window() / envelope() / statistics() / duration_s()`;`ForcePlotsPanel`
|
||||
只认这个接口,因此同一套浏览逻辑既能看内存会话也能看磁盘录制。
|
||||
|
||||
回放窗口在曲线被降采样时会明确写出「显示 N / 共 M 行」,而不是让操作员自己数点。
|
||||
|
||||
> 未做的验收项:真实分辨率与目标 ARM 板上的最大 RSS、积压与丢失测试(需要硬件),
|
||||
> 以及选中时间段的 CSV 导出。
|
||||
|
||||
---
|
||||
|
||||
## 9. 尚未完成(第三阶段及以后)
|
||||
|
||||
- PyOpenGL 三维显示(`mesh3d_panel.py` 目前仍是占位符,不是已有实现)。
|
||||
- CSV 的「导出选中时间段」(整段导出已随录制落盘;区间导出待做)。
|
||||
- 标定的稳定后采样(当前按固定时长采样,稳定性以 σ 呈现)。
|
||||
- 在目标 ARM 板上用真实分辨率测长时录制的最大 RSS、队列积压与丢失。
|
||||
|
||||
|
||||
Reference in New Issue
Block a user