orisys-app/docs/architecture/studio-stream-pipeline-migration.md

Studio acquisition on the SDK's concurrent pipeline

Historical migration record, 2026-10-01; moved to its product owner on 2026-10-19. Numbers, failures and proposed SDK changes below describe that run, not current acceptance. The original scope changed App workers and read the SDK; no SDK change was part of it. Current implementation owners are the SDK worker, finger worker and packet mapping. The open contact-detection and live-camera questions were not revalidated by this move.

What changed

The box/mini worker (shared/src/orisys_app_shared/acquisition/sdk_worker.py) replaced its get_img() → process_chain() → read_forces_from_sensor() loop with StreamPipeline(sensor, producer, contact=…, include=…) and now consumes the finished FrameSnapshot. The finger worker (.../acquisition/finger_worker.py) replaced read_source() → process_frame() → process_chain(tactile) with a pipeline over runtime.tactile.raw_sensor, built the way runtime.pipeline() builds it; runtime.warmup(color=False) still runs first.

The worker's public contract is unchanged: pause()/resume()/paused, latest(), reset(), stop(), consume_end_reason(), phase, capture_status(), filter_state(), the record-channel sinks, and every SdkPacket field the UI reads. The record channel still sees every published frame: the pipeline's on_frame hands each snapshot to the worker, which builds one packet and emits it as the serial loop did. SdkPacket construction now lives in one place, models.from_snapshot(snapshot, sensor, …), rather than being spelled out field by field in each worker.

Snapshot / app fact	Packet field
seq	frame_id
timestamp, collect time (timestamp − age_ms)	timestamp, wall_t, mono_t
fps	fps
normal_force, shear_x, shear_y, *_raw	same
unit	force_unit
image → BGR	processed_frame
flow, depth	same
centroid + sensor centroid_found	contact (ContactObservation)
stage times	process_ms
app	session_id, valid, invalid_reason, stale_forces, force_calibrated, calibrated_axes, calibration_version, raw_available, capture_fps, capture_dropped

mono_t is the collect time for a live source and frame_id × frame period for a file, so a replay keeps the video's own timeline. include= asks only for what the app can display; flow and depth are both carried because the 3D pane lets the operator switch between them at runtime.

The filter decision: the SDK filters, the app does not

Studio already pushed its filter choice into the device (FilterSettings.apply_to → Sensor.configure) and read the filtered values back; no app-side force filter exists. The migration consumes the SDK's fields as they are: snapshot.normal_force/shear_* (filtered and calibrated) and snapshot.normal_force_raw/shear_*_raw (the unfiltered HHD integrator output). ForcePresentation (offsets, deadband, tare) is untouched and still display-only.

The packet diff decided this, not inspection. With the SDK filter chain off, the two paths agree exactly on all six force fields — only possible if neither side filters separately. With median(3) + lowpass(10 Hz) pushed into the device, the raw six still agree exactly while the filtered three do not — and running the same path twice on the same clip reproduces that divergence (max |Δnormal_force| ≈ 0.010 N, 0 at the raw fields). The SDK's _update_force_sample_rate re-estimates the low-pass sample rate from wall-clock cadence, so a filtered value on a file source is a function of throughput as well as of the frames. A filtered force on a replay is reproducible on neither path, and the honest comparison is the raw one.

Packet diff

Same clip through both paths, one SdkPacket per frame, every field compared.

Run	Frames	Fields	Mismatches
box	26	27	none; contact only differs because the serial loop stated none
finger	32	29	one frame's contact detection
box, 264-frame clip	252	27	none; the same contact difference
finger, 264-frame clip	263	29	23 frames' contact detection

Tolerances: forces and *_raw at 1e-6, arrays at 1e-6 max absolute difference, ids/units/flags/provenance exactly equal. fps_present, valid_mask_matches_field and contact_frame_id are consistency checks. The only real difference is the contact observation the serial loop never stated: on 23 of 263 finger frames the new path finds a contact (a centre around (150, 250) in UV) where the serial path finds none. The serial path computes optical flow twice per frame (compute_deformation calls compute_flow after get_img already did), so its expansion mask and the pipeline's are not the same measurement. Which is right is unresolved.

Performance, measured

264-frame 1920×1080 box clip and the same frames through the finger runtime, two repetitions each, this machine.

Path	Wall (median)	Settled fps	process_ms
box, serial (252 frames)	9.42 s	51.9	16.5
box, pipeline	7.40 s	93.9	24.7
finger, serial (263 frames)	6.31 s	47.9	14.1
finger, pipeline	2.56 s	130.5	18.6

Read the columns separately. The rate is what the pipeline buys — 1.8× for the box, 2.7× for the finger — and it is the number the rail shows. The wall-clock gain is smaller (1.27× box, 2.47× finger) because a file pass also pays a fixed cost for opening the device and filling the SDK's caches; on the 33-frame clip that cost dominated completely (2.25 s vs 2.27 s) while the rate still went 66.6 → 112.3 fps. process_ms rises because it is now the stage sum the SDK measures. A live camera cannot be exercised here; its gain should be the pipeline's own.

Behavioural checks

• Pause keeps the device open and publishes nothing. pause() holds the pipeline at PauseLevel.CAPTURE_ONLY: the read continues (exposure, white balance and frame rate stay converged) while flow and HHD are skipped and no snapshot is published. A frame already in flight is dropped rather than handed over. resume(reset=True) rebases the tracker, which keeps a CAPTURE_ONLY hold out of the force reading.
• A recording session is structurally unchanged. Packets are still built in order, one per published frame, from the acquisition thread; frame_id, mono_t and the per-frame provenance flags mean what they did.
• No orphaned threads. stop() joins all three pipeline threads. A new test (shared/tests/test_stream_pipeline_acquisition.py) asserts that orisys-pipeline-produce/-flow/-hhd are gone after stop(), that a stop/start cycle runs a second session, and that the packet mapping carries every UI field.
• End of input. A file's end is reported as ended exactly as before, and a source that yields nothing reports an error instead of waiting out the ten-second deadline.

Verification actually run

Gate	Result	Before
shared/tests	389: 386 passed, 1 skipped, 2 failed	383: 382 passed, 1 skipped, 2 failed
apps/studio/tests	375: 358 passed, 17 skipped, 0 failed	same
apps/autocal/tests	358 passed, 0 failed	same
apps/studio/tools/ui_check.py	600 checks, 0 failures	same
real-GL smoke	14: 13 passed, 1 failed	same
tools/lint_gate.py, changed files	3 findings, all at HEAD (S110 ×3)	same

The two shared failures are the documented wall-clock pair in test_worker_frame_rate.py: the fake camera delivers ~21.5 fps on this machine against the 30 fps its thresholds assume, and the serial path measures 21.5 fps in the same conditions, so both are limited by the fake device's own thread. The GL failure is the pre-existing test_an_automatic_colour_scale_follows_the_frame.

What was kept, and what was decided alone

The serial chain is kept behind use_stream_pipeline=False on both workers; it is the reference the packet diff runs against, not the path a session uses. Decided here, not in the brief:

1. The finger pipeline is constructed directly, not through runtime.pipeline(), which hardcodes grab_and_process as the producer and so cannot see the source frame the record channel's pre-roll history wants, nor skip a repeated frame.
2. The producer holds the read until the capture source has published another frame, for a live camera. The serial loop did this through the source's sequence counter, and without it a 30 fps device is recorded at the polling rate with every measurement duplicated.
3. A file pass is paced by the declared frame period. Without it a replay runs as fast as the CPU allows and stamps 122 s of video time in 0.4 s of wall clock.
4. The SDK's contact centroid_found decides whether there is a contact, not the centre itself: the SDK leaves centroid at (0, 0) when it found nothing, and reporting that would put a marker in the frame's corner that nothing measured.
5. capture_dropped prefers the capture source's own counter, falling back to the pipeline's produced-minus-published backlog, which is all a file source can answer.
6. The packet's fps is measured at the collection times, the same basis the serial loop used, so a replay's reported rate means the same thing on both paths.

Left open: the finger contact-detection difference, and a live-camera measurement of the throughput gain.

Proposed SDK changes

The migration needed no SDK change, but three would remove workarounds the app carries.

1. Let a caller supply the producer to runtime.pipeline() (or accept a raw_frame=True / frame-sink option), so a finger caller gets the pre-unroll source frame and repeated-frame suppression without building the StreamPipeline itself. for_finger_runtime hardcodes grab_and_process today.
2. Pin the force low-pass sample rate for file sources instead of re-estimating it from wall-clock cadence (_update_force_sample_rate, sensor.py:814-844). _lowpass_rate_auto is already off for external-frame sources; the same for an opened video would make a filtered force reproducible, which is now false on both acquisition paths.
3. Expose the frames a pipeline has produced and published, and a documented way to wait for the next delivered frame. The app reaches into StreamPipeline._done and re-implements the repeat check around CaptureSource, because wait() is latest-wins and the only every-frame hook is on_frame, which the app must then drain itself for ordering and backpressure.