orisys-app/docs/architecture/studio-stream-pipeline-migration.md

# Studio acquisition on the SDK's concurrent pipeline

Historical migration record, 2026-10-01; moved to its product owner on 2026-10-19.
Numbers, failures and proposed SDK changes below describe that run, not current acceptance.
The original scope changed App workers and read the SDK; no SDK change was part of it.
Current implementation owners are the [SDK worker](../../shared/src/orisys_app_shared/acquisition/sdk_worker.py),
[finger worker](../../shared/src/orisys_app_shared/acquisition/finger_worker.py) and
[packet mapping](../../shared/src/orisys_app_shared/acquisition/models.py).
The open contact-detection and live-camera questions were not revalidated by this move.

## What changed

The box/mini worker (`shared/src/orisys_app_shared/acquisition/sdk_worker.py`) replaced
its `get_img()` → `process_chain()` → `read_forces_from_sensor()` loop with
`StreamPipeline(sensor, producer, contact=…, include=…)` and now consumes the finished
`FrameSnapshot`. The finger worker (`.../acquisition/finger_worker.py`) replaced
`read_source()` → `process_frame()` → `process_chain(tactile)` with a pipeline over
`runtime.tactile.raw_sensor`, built the way `runtime.pipeline()` builds it;
`runtime.warmup(color=False)` still runs first.

The worker's public contract is unchanged: `pause()/resume()/paused`, `latest()`,
`reset()`, `stop()`, `consume_end_reason()`, `phase`, `capture_status()`,
`filter_state()`, the record-channel sinks, and every `SdkPacket` field the UI reads.
The record channel still sees every published frame: the pipeline's `on_frame` hands
each snapshot to the worker, which builds one packet and emits it as the serial loop
did. `SdkPacket` construction now lives in one place,
`models.from_snapshot(snapshot, sensor, …)`, rather than being spelled out field by
field in each worker.

| Snapshot / app fact | Packet field |
|---|---|
| `seq` | `frame_id` |
| `timestamp`, collect time (`timestamp − age_ms`) | `timestamp`, `wall_t`, `mono_t` |
| `fps` | `fps` |
| `normal_force`, `shear_x`, `shear_y`, `*_raw` | same |
| `unit` | `force_unit` |
| `image` → BGR | `processed_frame` |
| `flow`, `depth` | same |
| `centroid` + sensor `centroid_found` | `contact` (`ContactObservation`) |
| stage times | `process_ms` |
| app | `session_id`, `valid`, `invalid_reason`, `stale_forces`, `force_calibrated`, `calibrated_axes`, `calibration_version`, `raw_available`, `capture_fps`, `capture_dropped` |

`mono_t` is the collect time for a live source and `frame_id × frame period` for a file,
so a replay keeps the video's own timeline. `include=` asks only for what the app can
display; flow and depth are both carried because the 3D pane lets the operator switch
between them at runtime.

## The filter decision: the SDK filters, the app does not

Studio already pushed its filter choice into the device (`FilterSettings.apply_to` →
`Sensor.configure`) and read the filtered values back; no app-side force filter exists.
The migration consumes the SDK's fields as they are: `snapshot.normal_force`/`shear_*`
(filtered and calibrated) and `snapshot.normal_force_raw`/`shear_*_raw` (the unfiltered
HHD integrator output). `ForcePresentation` (offsets, deadband, tare) is untouched and
still display-only.

The packet diff decided this, not inspection. With the SDK filter chain off, the two
paths agree **exactly** on all six force fields — only possible if neither side filters
separately. With `median(3) + lowpass(10 Hz)` pushed into the device, the *raw* six
still agree exactly while the filtered three do not — and running the **same path
twice** on the same clip reproduces that divergence (max |Δnormal_force| ≈ 0.010 N,
0 at the raw fields). The SDK's `_update_force_sample_rate` re-estimates the low-pass
sample rate from wall-clock cadence, so a *filtered* value on a file source is a
function of throughput as well as of the frames. A filtered force on a replay is
reproducible on neither path, and the honest comparison is the raw one.

## Packet diff

Same clip through both paths, one `SdkPacket` per frame, every field compared.

| Run | Frames | Fields | Mismatches |
|---|---|---|---|
| box | 26 | 27 | none; `contact` only differs because the serial loop stated none |
| finger | 32 | 29 | one frame's contact detection |
| box, 264-frame clip | 252 | 27 | none; the same `contact` difference |
| finger, 264-frame clip | 263 | 29 | 23 frames' contact detection |

Tolerances: forces and `*_raw` at 1e-6, arrays at 1e-6 max absolute difference,
ids/units/flags/provenance exactly equal. `fps_present`, `valid_mask_matches_field` and
`contact_frame_id` are consistency checks. The only real difference is the `contact`
observation the serial loop never stated: on 23 of 263 finger frames the new path finds
a contact (a centre around (150, 250) in UV) where the serial path finds none. The
serial path computes optical flow **twice per frame** (`compute_deformation` calls
`compute_flow` after `get_img` already did), so its expansion mask and the pipeline's
are not the same measurement. Which is right is unresolved.

## Performance, measured

264-frame 1920×1080 box clip and the same frames through the finger runtime, two
repetitions each, this machine.

| Path | Wall (median) | Settled fps | `process_ms` |
|---|---|---|---|
| box, serial (252 frames) | 9.42 s | 51.9 | 16.5 |
| box, pipeline | 7.40 s | 93.9 | 24.7 |
| finger, serial (263 frames) | 6.31 s | 47.9 | 14.1 |
| finger, pipeline | 2.56 s | 130.5 | 18.6 |

Read the columns separately. The rate is what the pipeline buys — **1.8×** for the box,
**2.7×** for the finger — and it is the number the rail shows. The wall-clock gain is
smaller (1.27× box, 2.47× finger) because a file pass also pays a fixed cost for
opening the device and filling the SDK's caches; on the 33-frame clip that cost
dominated completely (2.25 s vs 2.27 s) while the rate still went 66.6 → 112.3 fps.
`process_ms` rises because it is now the **stage sum** the SDK measures. A live camera
cannot be exercised here; its gain should be the pipeline's own.

## Behavioural checks

- **Pause keeps the device open and publishes nothing.** `pause()` holds the pipeline at
  `PauseLevel.CAPTURE_ONLY`: the read continues (exposure, white balance and frame rate
  stay converged) while flow and HHD are skipped and no snapshot is published. A frame
  already in flight is dropped rather than handed over. `resume(reset=True)` rebases
  the tracker, which keeps a `CAPTURE_ONLY` hold out of the force reading.
- **A recording session is structurally unchanged.** Packets are still built in order,
  one per published frame, from the acquisition thread; `frame_id`, `mono_t` and the
  per-frame provenance flags mean what they did.
- **No orphaned threads.** `stop()` joins all three pipeline threads. A new test
  (`shared/tests/test_stream_pipeline_acquisition.py`) asserts that
  `orisys-pipeline-produce/-flow/-hhd` are gone after `stop()`, that a stop/start cycle
  runs a second session, and that the packet mapping carries every UI field.
- **End of input.** A file's end is reported as `ended` exactly as before, and a source
  that yields nothing reports an error instead of waiting out the ten-second deadline.

## Verification actually run

| Gate | Result | Before |
|---|---|---|
| `shared/tests` | 389: 386 passed, 1 skipped, 2 failed | 383: 382 passed, 1 skipped, 2 failed |
| `apps/studio/tests` | 375: 358 passed, 17 skipped, 0 failed | same |
| `apps/autocal/tests` | 358 passed, 0 failed | same |
| `apps/studio/tools/ui_check.py` | 600 checks, 0 failures | same |
| real-GL smoke | 14: 13 passed, 1 failed | same |
| `tools/lint_gate.py`, changed files | 3 findings, all at HEAD (S110 ×3) | same |

The two shared failures are the documented wall-clock pair in
`test_worker_frame_rate.py`: the fake camera delivers ~21.5 fps on this machine against
the 30 fps its thresholds assume, and the serial path measures 21.5 fps in the same
conditions, so both are limited by the fake device's own thread. The GL failure is the
pre-existing `test_an_automatic_colour_scale_follows_the_frame`.

## What was kept, and what was decided alone

The serial chain is **kept** behind `use_stream_pipeline=False` on both workers; it is
the reference the packet diff runs against, not the path a session uses. Decided here,
not in the brief:

1. **The finger pipeline is constructed directly**, not through `runtime.pipeline()`,
   which hardcodes `grab_and_process` as the producer and so cannot see the source
   frame the record channel's pre-roll history wants, nor skip a repeated frame.
2. **The producer holds the read until the capture source has published another frame**,
   for a live camera. The serial loop did this through the source's sequence counter,
   and without it a 30 fps device is recorded at the polling rate with every
   measurement duplicated.
3. **A file pass is paced by the declared frame period.** Without it a replay runs as
   fast as the CPU allows and stamps 122 s of video time in 0.4 s of wall clock.
4. **The SDK's contact `centroid_found` decides whether there is a contact**, not the
   centre itself: the SDK leaves `centroid` at `(0, 0)` when it found nothing, and
   reporting that would put a marker in the frame's corner that nothing measured.
5. **`capture_dropped` prefers the capture source's own counter**, falling back to the
   pipeline's produced-minus-published backlog, which is all a file source can answer.
6. **The packet's `fps` is measured at the collection times**, the same basis the serial
   loop used, so a replay's reported rate means the same thing on both paths.

Left open: the finger contact-detection difference, and a live-camera measurement of the
throughput gain.

## Proposed SDK changes

The migration needed no SDK change, but three would remove workarounds the app carries.

1. **Let a caller supply the producer to `runtime.pipeline()`** (or accept a
   `raw_frame=True` / frame-sink option), so a finger caller gets the pre-unroll source
   frame and repeated-frame suppression without building the `StreamPipeline` itself.
   `for_finger_runtime` hardcodes `grab_and_process` today.
2. **Pin the force low-pass sample rate for file sources** instead of re-estimating it
   from wall-clock cadence (`_update_force_sample_rate`, `sensor.py`:814-844).
   `_lowpass_rate_auto` is already off for external-frame sources; the same for an
   opened video would make a filtered force reproducible, which is now false on both
   acquisition paths.
3. **Expose the frames a pipeline has produced and published**, and a documented way to
   wait for the next delivered frame. The app reaches into `StreamPipeline._done` and
   re-implements the repeat check around `CaptureSource`, because `wait()` is
   latest-wins and the only every-frame hook is `on_frame`, which the app must then
   drain itself for ordering and backpressure.