# Studio acquisition on the SDK's concurrent pipeline Historical migration record, 2026-10-01; moved to its product owner on 2026-10-19. Numbers, failures and proposed SDK changes below describe that run, not current acceptance. The original scope changed App workers and read the SDK; no SDK change was part of it. Current implementation owners are the [SDK worker](../../shared/src/orisys_app_shared/acquisition/sdk_worker.py), [finger worker](../../shared/src/orisys_app_shared/acquisition/finger_worker.py) and [packet mapping](../../shared/src/orisys_app_shared/acquisition/models.py). The open contact-detection and live-camera questions were not revalidated by this move. ## What changed The box/mini worker (`shared/src/orisys_app_shared/acquisition/sdk_worker.py`) replaced its `get_img()` → `process_chain()` → `read_forces_from_sensor()` loop with `StreamPipeline(sensor, producer, contact=…, include=…)` and now consumes the finished `FrameSnapshot`. The finger worker (`.../acquisition/finger_worker.py`) replaced `read_source()` → `process_frame()` → `process_chain(tactile)` with a pipeline over `runtime.tactile.raw_sensor`, built the way `runtime.pipeline()` builds it; `runtime.warmup(color=False)` still runs first. The worker's public contract is unchanged: `pause()/resume()/paused`, `latest()`, `reset()`, `stop()`, `consume_end_reason()`, `phase`, `capture_status()`, `filter_state()`, the record-channel sinks, and every `SdkPacket` field the UI reads. The record channel still sees every published frame: the pipeline's `on_frame` hands each snapshot to the worker, which builds one packet and emits it as the serial loop did. `SdkPacket` construction now lives in one place, `models.from_snapshot(snapshot, sensor, …)`, rather than being spelled out field by field in each worker. | Snapshot / app fact | Packet field | |---|---| | `seq` | `frame_id` | | `timestamp`, collect time (`timestamp − age_ms`) | `timestamp`, `wall_t`, `mono_t` | | `fps` | `fps` | | `normal_force`, `shear_x`, `shear_y`, `*_raw` | same | | `unit` | `force_unit` | | `image` → BGR | `processed_frame` | | `flow`, `depth` | same | | `centroid` + sensor `centroid_found` | `contact` (`ContactObservation`) | | stage times | `process_ms` | | app | `session_id`, `valid`, `invalid_reason`, `stale_forces`, `force_calibrated`, `calibrated_axes`, `calibration_version`, `raw_available`, `capture_fps`, `capture_dropped` | `mono_t` is the collect time for a live source and `frame_id × frame period` for a file, so a replay keeps the video's own timeline. `include=` asks only for what the app can display; flow and depth are both carried because the 3D pane lets the operator switch between them at runtime. ## The filter decision: the SDK filters, the app does not Studio already pushed its filter choice into the device (`FilterSettings.apply_to` → `Sensor.configure`) and read the filtered values back; no app-side force filter exists. The migration consumes the SDK's fields as they are: `snapshot.normal_force`/`shear_*` (filtered and calibrated) and `snapshot.normal_force_raw`/`shear_*_raw` (the unfiltered HHD integrator output). `ForcePresentation` (offsets, deadband, tare) is untouched and still display-only. The packet diff decided this, not inspection. With the SDK filter chain off, the two paths agree **exactly** on all six force fields — only possible if neither side filters separately. With `median(3) + lowpass(10 Hz)` pushed into the device, the *raw* six still agree exactly while the filtered three do not — and running the **same path twice** on the same clip reproduces that divergence (max |Δnormal_force| ≈ 0.010 N, 0 at the raw fields). The SDK's `_update_force_sample_rate` re-estimates the low-pass sample rate from wall-clock cadence, so a *filtered* value on a file source is a function of throughput as well as of the frames. A filtered force on a replay is reproducible on neither path, and the honest comparison is the raw one. ## Packet diff Same clip through both paths, one `SdkPacket` per frame, every field compared. | Run | Frames | Fields | Mismatches | |---|---|---|---| | box | 26 | 27 | none; `contact` only differs because the serial loop stated none | | finger | 32 | 29 | one frame's contact detection | | box, 264-frame clip | 252 | 27 | none; the same `contact` difference | | finger, 264-frame clip | 263 | 29 | 23 frames' contact detection | Tolerances: forces and `*_raw` at 1e-6, arrays at 1e-6 max absolute difference, ids/units/flags/provenance exactly equal. `fps_present`, `valid_mask_matches_field` and `contact_frame_id` are consistency checks. The only real difference is the `contact` observation the serial loop never stated: on 23 of 263 finger frames the new path finds a contact (a centre around (150, 250) in UV) where the serial path finds none. The serial path computes optical flow **twice per frame** (`compute_deformation` calls `compute_flow` after `get_img` already did), so its expansion mask and the pipeline's are not the same measurement. Which is right is unresolved. ## Performance, measured 264-frame 1920×1080 box clip and the same frames through the finger runtime, two repetitions each, this machine. | Path | Wall (median) | Settled fps | `process_ms` | |---|---|---|---| | box, serial (252 frames) | 9.42 s | 51.9 | 16.5 | | box, pipeline | 7.40 s | 93.9 | 24.7 | | finger, serial (263 frames) | 6.31 s | 47.9 | 14.1 | | finger, pipeline | 2.56 s | 130.5 | 18.6 | Read the columns separately. The rate is what the pipeline buys — **1.8×** for the box, **2.7×** for the finger — and it is the number the rail shows. The wall-clock gain is smaller (1.27× box, 2.47× finger) because a file pass also pays a fixed cost for opening the device and filling the SDK's caches; on the 33-frame clip that cost dominated completely (2.25 s vs 2.27 s) while the rate still went 66.6 → 112.3 fps. `process_ms` rises because it is now the **stage sum** the SDK measures. A live camera cannot be exercised here; its gain should be the pipeline's own. ## Behavioural checks - **Pause keeps the device open and publishes nothing.** `pause()` holds the pipeline at `PauseLevel.CAPTURE_ONLY`: the read continues (exposure, white balance and frame rate stay converged) while flow and HHD are skipped and no snapshot is published. A frame already in flight is dropped rather than handed over. `resume(reset=True)` rebases the tracker, which keeps a `CAPTURE_ONLY` hold out of the force reading. - **A recording session is structurally unchanged.** Packets are still built in order, one per published frame, from the acquisition thread; `frame_id`, `mono_t` and the per-frame provenance flags mean what they did. - **No orphaned threads.** `stop()` joins all three pipeline threads. A new test (`shared/tests/test_stream_pipeline_acquisition.py`) asserts that `orisys-pipeline-produce/-flow/-hhd` are gone after `stop()`, that a stop/start cycle runs a second session, and that the packet mapping carries every UI field. - **End of input.** A file's end is reported as `ended` exactly as before, and a source that yields nothing reports an error instead of waiting out the ten-second deadline. ## Verification actually run | Gate | Result | Before | |---|---|---| | `shared/tests` | 389: 386 passed, 1 skipped, 2 failed | 383: 382 passed, 1 skipped, 2 failed | | `apps/studio/tests` | 375: 358 passed, 17 skipped, 0 failed | same | | `apps/autocal/tests` | 358 passed, 0 failed | same | | `apps/studio/tools/ui_check.py` | 600 checks, 0 failures | same | | real-GL smoke | 14: 13 passed, 1 failed | same | | `tools/lint_gate.py`, changed files | 3 findings, all at HEAD (S110 ×3) | same | The two shared failures are the documented wall-clock pair in `test_worker_frame_rate.py`: the fake camera delivers ~21.5 fps on this machine against the 30 fps its thresholds assume, and the serial path measures 21.5 fps in the same conditions, so both are limited by the fake device's own thread. The GL failure is the pre-existing `test_an_automatic_colour_scale_follows_the_frame`. ## What was kept, and what was decided alone The serial chain is **kept** behind `use_stream_pipeline=False` on both workers; it is the reference the packet diff runs against, not the path a session uses. Decided here, not in the brief: 1. **The finger pipeline is constructed directly**, not through `runtime.pipeline()`, which hardcodes `grab_and_process` as the producer and so cannot see the source frame the record channel's pre-roll history wants, nor skip a repeated frame. 2. **The producer holds the read until the capture source has published another frame**, for a live camera. The serial loop did this through the source's sequence counter, and without it a 30 fps device is recorded at the polling rate with every measurement duplicated. 3. **A file pass is paced by the declared frame period.** Without it a replay runs as fast as the CPU allows and stamps 122 s of video time in 0.4 s of wall clock. 4. **The SDK's contact `centroid_found` decides whether there is a contact**, not the centre itself: the SDK leaves `centroid` at `(0, 0)` when it found nothing, and reporting that would put a marker in the frame's corner that nothing measured. 5. **`capture_dropped` prefers the capture source's own counter**, falling back to the pipeline's produced-minus-published backlog, which is all a file source can answer. 6. **The packet's `fps` is measured at the collection times**, the same basis the serial loop used, so a replay's reported rate means the same thing on both paths. Left open: the finger contact-detection difference, and a live-camera measurement of the throughput gain. ## Proposed SDK changes The migration needed no SDK change, but three would remove workarounds the app carries. 1. **Let a caller supply the producer to `runtime.pipeline()`** (or accept a `raw_frame=True` / frame-sink option), so a finger caller gets the pre-unroll source frame and repeated-frame suppression without building the `StreamPipeline` itself. `for_finger_runtime` hardcodes `grab_and_process` today. 2. **Pin the force low-pass sample rate for file sources** instead of re-estimating it from wall-clock cadence (`_update_force_sample_rate`, `sensor.py`:814-844). `_lowpass_rate_auto` is already off for external-frame sources; the same for an opened video would make a filtered force reproducible, which is now false on both acquisition paths. 3. **Expose the frames a pipeline has produced and published**, and a documented way to wait for the next delivered frame. The app reaches into `StreamPipeline._done` and re-implements the repeat check around `CaptureSource`, because `wait()` is latest-wins and the only every-frame hook is `on_frame`, which the app must then drain itself for ordering and backpressure.