From fbf32b5303357123904d5ecdf76d8403459844fb Mon Sep 17 00:00:00 2001 From: ykcheungab Date: Sun, 4 Oct 2026 14:13:05 +0800 Subject: [PATCH] Serve replay playback from a cached analysis result Replay re-ran the SDK chain on every seek; Part A's pass stores the flow samples once. Opening a recording now reuses a compatible result or runs the pass behind a progress/cancel surface, and one cursor with two modes (recorded CSV, analysed) feeds the charts, the 2D canvas and the 3D pane by frame id. A threshold, colour or idle-dot change redraws from the stored frames and writes nothing; a cancelled pass is reported and never played. Verified: apps/studio/tests 440 items / 0 failed / 18 skipped (15 new, no existing assertion changed); shared 417 / 0 / 1; autocal 358. --- apps/studio/docs/replay-panel.md | 165 +++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 apps/studio/docs/replay-panel.md diff --git a/apps/studio/docs/replay-panel.md b/apps/studio/docs/replay-panel.md new file mode 100644 index 0000000..ff80496 --- /dev/null +++ b/apps/studio/docs/replay-panel.md @@ -0,0 +1,165 @@ +# Replay panel contract + +The replay panel plays one recording — or one video — against a single cursor. Each +capability states its requirement, data source and a falsifying check. The result format and +playback semantics are owned by Part A ([replay-analysis.md](replay-analysis.md)) and Part B +([replay-analysis-part-b.md](replay-analysis-part-b.md)). + +Status: Parts A and B shipped at `cf49b08`; the depth channel — result format and 2D view — is +uncommitted; the cache-root and lifetime policy and the display-control separation (Part C) +are decided below, not implemented. + +## 1. Load a video + +Selecting a video always gives a playable timeline, and the panel says what the file is. +`load_video` calls a folder holding `forces.csv` beside the video a recording; anything else +opens as a bare video (`VideoOnlySource`) and plays, with `playback.video_only_opened` on the +rail and `playback.video_only_notice` in the notes. A file that cannot be decoded is refused +with its reason. + +Not available for a bare video: force rows (the charts are empty by construction), +`session.json` metadata, and an analysis pass — `recording_directory()` needs a +`session.json` and reports `playback.analysis_no_source`; the only route to numbers is +rerunning the acquisition pipeline over the video (`on_recompute_toggled`), forward only. + +Source: `record/playback.py`, `replay_controller.py`. +Verify: `test_ui_smoke.py::test_a_bare_video_plays_even_though_it_is_not_a_recording`; +`test_replay_controller.py::test_a_bare_video_is_opened_as_a_video_and_refused_outside_the_view`. + +## 2. Play and compute on the loaded video + +Opening a recording either serves a compatible stored result or runs one pass behind a +progress-and-cancel surface. `on_analysis_requested` asks +`AnalysisService.stored_result()` first — a file read, the surface showing +`analysis.reused` — and otherwise starts `AnalysisWorker`, a plain thread whose `Progress` is +coalesced at 0.1 s, with `AnalysisProgress` showing written/expected frames and elapsed time. +The timeline stays in `recording` mode until the pass commits: a half-written result never +reaches the screen. + +The pass is sequential and forward-only: `_pass_frames` walks segments and frames in order, +and the chain is stateful (`Sensor.reset()` clears the tracker). Cancelling sets an event; the +result keeps its `INCOMPLETE` marker and default readers refuse it (Part A §7). Measured: +0.48–0.51 wall-seconds per video-second (Part A §8: 0.5036 s mean against 0.5052 s for the bare +chain) — ten minutes is ≈4.8 minutes. + +Source: `analysis/service.py`, `analysis_worker.py`, `ui/analysis_progress.py`. +Verify: `test_replay_analysis.py::test_opening_a_recording_with_a_stored_result_runs_no_pass`, +`::test_opening_a_recording_without_a_result_starts_the_pass`, +`::test_the_worker_reports_progress_and_a_cancelled_outcome`. + +## 3. Cursor playback + +The cursor is one position for the whole panel, on the recording's own signal time: `t_rel_s` +from `frames.csv`, the axis `ReplayEngine.position()` and `AnalysisReader.frame_at()` index by. +Container time is never derived from it or converted into it. Measured: 33 container frames at +30 fps are 1.100 s of video and 1.920 s of signal at 16.664 fps (Part A §5). + +Frames are addressed by the recording's own frame id, never an index: `AnalysisSession.frame` +refuses an id the result does not hold (`AnalysisNotFoundError`), and the evidence recording's +ids run 20…52, so an index-based path would answer with another frame's measurement. A seek +resolves to the stored frame at or before the position — the frame sequential playback passes — +and re-runs no pass. + +Source: `replay.py`, `replay_analysis.py`. +Verify: `test_replay_analysis.py::test_position_is_signal_time_and_resolves_to_a_stored_frame_id`, +`::test_a_frame_id_the_recording_does_not_hold_is_refused`, +`::test_a_seek_equals_playing_forward_to_the_same_frame`. + +## 4. Synchronised charts, 2D view and 3D view + +One cursor drives all three. + +| View | Data source | +|---|---| +| Charts | the recorded `forces.csv` (`RecordingCurveSource`); the result stores no force | +| 2D canvas | the cached per-frame samples and depth, drawn at display time | +| 3D pane | the packet built from the cached frame (`AnalysisSession.packet` → `Scene3DView.submit_packet`) | + +A seek equals playing forward to that frame: the cursor resolves to one stored frame id and all +three views are fed from it. A derived view chosen while the timeline reads the video is drawn +from the result at the same cursor (`available_result`). Not synchronised: the 3D pane's depth +(`AnalysisSession.packet` sets `depth=None`, `depth_valid=False`); forces (the packet carries +`forces_valid=False` and zero, so the pane's force readout is not a measurement here); the 3D +pane with no result, which nothing feeds; and a recompute pass, where the pipeline owns the +timeline (`on_cursor_moved`). + +Display settings are reads: the threshold re-derives activity from the stored `magnitude`, colour +range and idle dots live in the draw call, and a grid-moving setting is refused by name +(`GRID_KEYS`), the frame still drawn from the recorded grid. +Verify: `test_replay_analysis.py::test_the_3d_pane_is_fed_the_same_frame_the_canvas_draws`, +`::test_a_display_change_that_moves_the_grid_is_refused_not_served`. + +## 5. What the 2D view can show + +The modes come from the frame's own fields and from whether a result exists (`plan_image`, +`result_available`): `arrows_pure` (Arrows), `arrows_stitched` (Arrows / Captured), `depth` +(Depth map), `recorded` (Recorded image). With a result all four are offered, even on the +recording's timeline, defaulting to `recorded`; with none, `recorded` alone plus +`panels.replay_raw_note`. + +| Mode | Drawn from | +|---|---| +| recorded | the decoded `raw_*.mp4` frame (`RecordingReader.video_frame_at`) | +| arrows / arrows-captured | the cached samples: the grid keyframe's locations plus this frame's float32 `dx`/`dy`/`magnitude` and bit-packed `active` (4,034 B/frame before the depth channel; 4,048 B/frame now) | +| depth | a per-frame depth flag, a non-zero count and min/max; only a non-empty map adds a uint8 plane quantised over that frame's range and zlib'd at level 6 (island restriction rejected: 418 B vs 486 B on the worst frame) | + +Measured: the depth map is exactly all zeros on 241 of 264 frames of the eight-times clip +(≈91 %; 32 of 33 on the box recording) and never exceeded 336 of 84,000 pixels (0.4 %). A blank +depth view is therefore usually correct and must be explainable: `panels.depth_empty` says the +frame's depth is all zeros, `panels.depth_unavailable` says the frame has no depth channel at +all — different facts, neither drawing the processed image. + +A mode needing derived data is served from the result, never faked: with none, that view is not +offered and `panels.replay_raw_note` names the action that produces a result. +Verify: `test_image_backend.py` (the plan), `test_replay_analysis.py::test_a_derived_mode_is_served_from_the_result_even_on_the_recording_timeline`, +`::test_the_empty_depth_note_is_not_the_missing_depth_note`, `test_analysis_depth.py::test_an_empty_map_costs_a_header_and_states_its_shape`. + +## Cache and export policies + +No rendered 2D images are cached. PNGs of the arrow and depth views would cost 180–540 MB per +ten minutes and bake display settings into pixels, destroying the property that a threshold, +colour or idle-dot change is a read. Drawing a frame from the cached numbers costs ≈1.4 ms, the +idle-dot pass 37 µs (owner's bench, not reproduced here). Check: +`::test_a_display_change_is_a_read_and_writes_no_result`, `::test_a_mode_change_writes_no_analysis`. + +Derived views cannot be computed from raw data during a scrub: the chain is stateful and +forward-only, so a backward seek would need a full re-run at 0.48–0.51 s per video-second; the +panel scrubs stored numbers and offers that run separately. + +Cache location. Results belong under an OS-correct derived-data root, +`%LOCALAPPDATA%\OrisysStudio\analysis-cache\` — never the user's recording folder or the app's +`outputs/`, which in the packaged build is the EXE's own directory and holds the recordings +(`paths.recordings_dir()` → `outputs/recordings`). The key is path-independent +(`recording_identity` ignores the path, includes session facts and segment sizes), so a copied +recording finds its result. Today `ResultCache.for_recording` roots the cache at +`/analysis`; the OS root is decided, not implemented. + +Cache lifetime. A size cap with LRU eviction (1 GiB, `DEFAULT_MAX_BYTES`), a startup sweep of +stale and incomplete results, a manual "clear analysis cache" action, and wipe-on-exit only as +an explicit preference that is off by default — because wiping on exit forces a full re-analysis +(≈4.8 minutes per ten minutes) on reopen and does not protect against a crash. Implemented +today: `evict()` removes whole results, incomplete first then oldest, protecting the caller's +`keep` — with no call site in the app, so there is no sweep, no clear action and no preference. + +Export. The deliberate way to keep a result is to export it: ≈70 MB per ten minutes, portable +because the key ignores where the file sits. A rendered arrow or depth video is a later +possibility, belonging to export rather than the cache. Not implemented. + +## Owners and status + +| Part | Owner | Status | +|---|---|---| +| Format, cache, reader, pass | `orisys_app_shared.analysis` | A/B shipped; depth channel uncommitted | +| Timeline, cursor, policy, pass thread, progress surface | `replay_analysis.py`, `replay.py`, `replay_controller.py`, `analysis_worker.py`, `ui/analysis_progress.py` | shipped; the depth view and the result-aware mode plan are uncommitted | +| Mode plan, drawing, window wiring | `image_backend.py`, `acquisition/arrows.py`, `app_window.py` | shipped (Part B §12 names the seams); the result-aware plan is uncommitted | +| Cache root and lifetime | `analysis/cache.py`, app preferences | decided, not implemented | +| Display-control separation: rail reset as the measurement reference, a separate display baseline, the F8 view reset | `toolbar.py`, `display_prefs.py`, `app_window.py` | not started; Part B §12 records these as unchanged. The pass accepts a `reference_frame` but `_analysis_service` passes none | + +## Not confirmed in code + +* The render cost (≈1.4 ms, idle dots 37 µs) and the PNG estimate (180–540 MB per ten minutes) + are the owner's bench, not measurements here; Part B §14's single-frame snapshots (4.2–4.9 kB) + suggest the estimate is high, so it should be re-measured. +* The OS cache root, startup sweep, clear action, wipe-on-exit preference and result export + have no code in `orisys-app`; they are decisions. +* The status lines describe the working tree, not `cf49b08`: the depth work is uncommitted.