From 8a027bfcb9a4d00630c0d5a05305b15e06331fa1 Mon Sep 17 00:00:00 2001 From: ykcheungab Date: Mon, 19 Oct 2026 10:15:45 +0800 Subject: [PATCH] =?UTF-8?q?=E6=8E=A5=E7=AE=A1=20Studio=20=E6=B5=81?= =?UTF-8?q?=E6=B0=B4=E7=BA=BF=E8=BF=81=E7=A7=BB=E8=AE=B0=E5=BD=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将历史比较放回产品仓库并标明现行实现所有者。文档与仓库门禁通过，原记录正文保留。 --- .../studio-stream-pipeline-migration.md | 184 ++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 docs/architecture/studio-stream-pipeline-migration.md diff --git a/docs/architecture/studio-stream-pipeline-migration.md b/docs/architecture/studio-stream-pipeline-migration.md new file mode 100644 index 0000000..4bd066b --- /dev/null +++ b/docs/architecture/studio-stream-pipeline-migration.md @@ -0,0 +1,184 @@ +# Studio acquisition on the SDK's concurrent pipeline + +Historical migration record, 2026-10-01; moved to its product owner on 2026-10-19. +Numbers, failures and proposed SDK changes below describe that run, not current acceptance. +The original scope changed App workers and read the SDK; no SDK change was part of it. +Current implementation owners are the [SDK worker](../../shared/src/orisys_app_shared/acquisition/sdk_worker.py), +[finger worker](../../shared/src/orisys_app_shared/acquisition/finger_worker.py) and +[packet mapping](../../shared/src/orisys_app_shared/acquisition/models.py). +The open contact-detection and live-camera questions were not revalidated by this move. + +## What changed + +The box/mini worker (`shared/src/orisys_app_shared/acquisition/sdk_worker.py`) replaced +its `get_img()` → `process_chain()` → `read_forces_from_sensor()` loop with +`StreamPipeline(sensor, producer, contact=…, include=…)` and now consumes the finished +`FrameSnapshot`. The finger worker (`.../acquisition/finger_worker.py`) replaced +`read_source()` → `process_frame()` → `process_chain(tactile)` with a pipeline over +`runtime.tactile.raw_sensor`, built the way `runtime.pipeline()` builds it; +`runtime.warmup(color=False)` still runs first. + +The worker's public contract is unchanged: `pause()/resume()/paused`, `latest()`, +`reset()`, `stop()`, `consume_end_reason()`, `phase`, `capture_status()`, +`filter_state()`, the record-channel sinks, and every `SdkPacket` field the UI reads. +The record channel still sees every published frame: the pipeline's `on_frame` hands +each snapshot to the worker, which builds one packet and emits it as the serial loop +did. `SdkPacket` construction now lives in one place, +`models.from_snapshot(snapshot, sensor, …)`, rather than being spelled out field by +field in each worker. + +| Snapshot / app fact | Packet field | +|---|---| +| `seq` | `frame_id` | +| `timestamp`, collect time (`timestamp − age_ms`) | `timestamp`, `wall_t`, `mono_t` | +| `fps` | `fps` | +| `normal_force`, `shear_x`, `shear_y`, `*_raw` | same | +| `unit` | `force_unit` | +| `image` → BGR | `processed_frame` | +| `flow`, `depth` | same | +| `centroid` + sensor `centroid_found` | `contact` (`ContactObservation`) | +| stage times | `process_ms` | +| app | `session_id`, `valid`, `invalid_reason`, `stale_forces`, `force_calibrated`, `calibrated_axes`, `calibration_version`, `raw_available`, `capture_fps`, `capture_dropped` | + +`mono_t` is the collect time for a live source and `frame_id × frame period` for a file, +so a replay keeps the video's own timeline. `include=` asks only for what the app can +display; flow and depth are both carried because the 3D pane lets the operator switch +between them at runtime. + +## The filter decision: the SDK filters, the app does not + +Studio already pushed its filter choice into the device (`FilterSettings.apply_to` → +`Sensor.configure`) and read the filtered values back; no app-side force filter exists. +The migration consumes the SDK's fields as they are: `snapshot.normal_force`/`shear_*` +(filtered and calibrated) and `snapshot.normal_force_raw`/`shear_*_raw` (the unfiltered +HHD integrator output). `ForcePresentation` (offsets, deadband, tare) is untouched and +still display-only. + +The packet diff decided this, not inspection. With the SDK filter chain off, the two +paths agree **exactly** on all six force fields — only possible if neither side filters +separately. With `median(3) + lowpass(10 Hz)` pushed into the device, the *raw* six +still agree exactly while the filtered three do not — and running the **same path +twice** on the same clip reproduces that divergence (max |Δnormal_force| ≈ 0.010 N, +0 at the raw fields). The SDK's `_update_force_sample_rate` re-estimates the low-pass +sample rate from wall-clock cadence, so a *filtered* value on a file source is a +function of throughput as well as of the frames. A filtered force on a replay is +reproducible on neither path, and the honest comparison is the raw one. + +## Packet diff + +Same clip through both paths, one `SdkPacket` per frame, every field compared. + +| Run | Frames | Fields | Mismatches | +|---|---|---|---| +| box | 26 | 27 | none; `contact` only differs because the serial loop stated none | +| finger | 32 | 29 | one frame's contact detection | +| box, 264-frame clip | 252 | 27 | none; the same `contact` difference | +| finger, 264-frame clip | 263 | 29 | 23 frames' contact detection | + +Tolerances: forces and `*_raw` at 1e-6, arrays at 1e-6 max absolute difference, +ids/units/flags/provenance exactly equal. `fps_present`, `valid_mask_matches_field` and +`contact_frame_id` are consistency checks. The only real difference is the `contact` +observation the serial loop never stated: on 23 of 263 finger frames the new path finds +a contact (a centre around (150, 250) in UV) where the serial path finds none. The +serial path computes optical flow **twice per frame** (`compute_deformation` calls +`compute_flow` after `get_img` already did), so its expansion mask and the pipeline's +are not the same measurement. Which is right is unresolved. + +## Performance, measured + +264-frame 1920×1080 box clip and the same frames through the finger runtime, two +repetitions each, this machine. + +| Path | Wall (median) | Settled fps | `process_ms` | +|---|---|---|---| +| box, serial (252 frames) | 9.42 s | 51.9 | 16.5 | +| box, pipeline | 7.40 s | 93.9 | 24.7 | +| finger, serial (263 frames) | 6.31 s | 47.9 | 14.1 | +| finger, pipeline | 2.56 s | 130.5 | 18.6 | + +Read the columns separately. The rate is what the pipeline buys — **1.8×** for the box, +**2.7×** for the finger — and it is the number the rail shows. The wall-clock gain is +smaller (1.27× box, 2.47× finger) because a file pass also pays a fixed cost for +opening the device and filling the SDK's caches; on the 33-frame clip that cost +dominated completely (2.25 s vs 2.27 s) while the rate still went 66.6 → 112.3 fps. +`process_ms` rises because it is now the **stage sum** the SDK measures. A live camera +cannot be exercised here; its gain should be the pipeline's own. + +## Behavioural checks + +- **Pause keeps the device open and publishes nothing.** `pause()` holds the pipeline at + `PauseLevel.CAPTURE_ONLY`: the read continues (exposure, white balance and frame rate + stay converged) while flow and HHD are skipped and no snapshot is published. A frame + already in flight is dropped rather than handed over. `resume(reset=True)` rebases + the tracker, which keeps a `CAPTURE_ONLY` hold out of the force reading. +- **A recording session is structurally unchanged.** Packets are still built in order, + one per published frame, from the acquisition thread; `frame_id`, `mono_t` and the + per-frame provenance flags mean what they did. +- **No orphaned threads.** `stop()` joins all three pipeline threads. A new test + (`shared/tests/test_stream_pipeline_acquisition.py`) asserts that + `orisys-pipeline-produce/-flow/-hhd` are gone after `stop()`, that a stop/start cycle + runs a second session, and that the packet mapping carries every UI field. +- **End of input.** A file's end is reported as `ended` exactly as before, and a source + that yields nothing reports an error instead of waiting out the ten-second deadline. + +## Verification actually run + +| Gate | Result | Before | +|---|---|---| +| `shared/tests` | 389: 386 passed, 1 skipped, 2 failed | 383: 382 passed, 1 skipped, 2 failed | +| `apps/studio/tests` | 375: 358 passed, 17 skipped, 0 failed | same | +| `apps/autocal/tests` | 358 passed, 0 failed | same | +| `apps/studio/tools/ui_check.py` | 600 checks, 0 failures | same | +| real-GL smoke | 14: 13 passed, 1 failed | same | +| `tools/lint_gate.py`, changed files | 3 findings, all at HEAD (S110 ×3) | same | + +The two shared failures are the documented wall-clock pair in +`test_worker_frame_rate.py`: the fake camera delivers ~21.5 fps on this machine against +the 30 fps its thresholds assume, and the serial path measures 21.5 fps in the same +conditions, so both are limited by the fake device's own thread. The GL failure is the +pre-existing `test_an_automatic_colour_scale_follows_the_frame`. + +## What was kept, and what was decided alone + +The serial chain is **kept** behind `use_stream_pipeline=False` on both workers; it is +the reference the packet diff runs against, not the path a session uses. Decided here, +not in the brief: + +1. **The finger pipeline is constructed directly**, not through `runtime.pipeline()`, + which hardcodes `grab_and_process` as the producer and so cannot see the source + frame the record channel's pre-roll history wants, nor skip a repeated frame. +2. **The producer holds the read until the capture source has published another frame**, + for a live camera. The serial loop did this through the source's sequence counter, + and without it a 30 fps device is recorded at the polling rate with every + measurement duplicated. +3. **A file pass is paced by the declared frame period.** Without it a replay runs as + fast as the CPU allows and stamps 122 s of video time in 0.4 s of wall clock. +4. **The SDK's contact `centroid_found` decides whether there is a contact**, not the + centre itself: the SDK leaves `centroid` at `(0, 0)` when it found nothing, and + reporting that would put a marker in the frame's corner that nothing measured. +5. **`capture_dropped` prefers the capture source's own counter**, falling back to the + pipeline's produced-minus-published backlog, which is all a file source can answer. +6. **The packet's `fps` is measured at the collection times**, the same basis the serial + loop used, so a replay's reported rate means the same thing on both paths. + +Left open: the finger contact-detection difference, and a live-camera measurement of the +throughput gain. + +## Proposed SDK changes + +The migration needed no SDK change, but three would remove workarounds the app carries. + +1. **Let a caller supply the producer to `runtime.pipeline()`** (or accept a + `raw_frame=True` / frame-sink option), so a finger caller gets the pre-unroll source + frame and repeated-frame suppression without building the `StreamPipeline` itself. + `for_finger_runtime` hardcodes `grab_and_process` today. +2. **Pin the force low-pass sample rate for file sources** instead of re-estimating it + from wall-clock cadence (`_update_force_sample_rate`, `sensor.py`:814-844). + `_lowpass_rate_auto` is already off for external-frame sources; the same for an + opened video would make a filtered force reproducible, which is now false on both + acquisition paths. +3. **Expose the frames a pipeline has produced and published**, and a documented way to + wait for the next delivered frame. The app reaches into `StreamPipeline._done` and + re-implements the repeat check around `CaptureSource`, because `wait()` is + latest-wins and the only every-frame hook is `on_frame`, which the app must then + drain itself for ordering and backpressure. \ No newline at end of file