orisys-sensor-design/docs/probe-method.md

# Reading the probe sensitivity report

This is the operator's guide to `sensitivity_report.json` and
`optimizer_report.json`. The interface itself is frozen in
[`probe-pipeline.md`](probe-pipeline.md); this document explains what the numbers
mean, what they do **not** mean, and how to use the optimizer.

The bench deforms the sensing surface through a known positive x/y/z sweep
(0.01 mm .. 0.10 mm in 0.01 mm steps) in a local probe-average basis, renders what
`cam_sensor` sees, and measures DIS optical flow against the zero-displacement
reference at two scales (the real render and the runtime's processed resolution).

```
deform -> render -> track (DIS flow) -> sensitivity (score + report) -> optimise
```

Every stage writes JSON under `build/probe/<sensor>/<revision>/`. The sensitivity
stage reads `render_manifest.json`, `flow_report.json` and `flow/<scale>/*.npy`
and writes `sensitivity_report.json`. It never re-runs a stage; use the CLI's
`--skip-deform/--skip-render/--skip-track` to reuse existing artifacts.

## The one thing to remember

**Sensitivity here is a relative ranking under a fixed estimator (DIS) and a
fixed camera. It is not force calibration.** The absolute `gain_mm_per_mm`
carries the measurement geometry with it, so only compare scores produced with the
same `radius_mm`, ROI, resolution and tracker settings.

## The floor is not a noise model (read this first)

**An earlier version of this document stated that the marker cell must match the
estimator's effective window, and that nx=31 was the best pattern. That is retracted.**
The conclusion was driven by `floor_px`, and `floor_px` is not a valid noise model.

`floor_px` is measured as `reference` vs a re-render at **seed + 1** — Cycles antialiasing
jitter, ~5 grey levels RMS on half the pixels, scaling with edge contrast. It cannot rank
patterns, for three measured reasons:

1. **Its trend reverses under additive noise.** p95 |flow| over the ROI, same DIS config:

   | pattern | render-seed | injected σ=2 | σ=4 |
   |---|---|---|---|
   | nx=80 | 0.0292 | **0.0161** | 0.0278 |
   | nx=31 | 0.0376 | **0.0084** | 0.0182 |
   | nx=56 | 0.0134 | 0.0102 | 0.0203 |

2. **It understates the real floor, coarse patterns most.** Measured on idle dd02 footage
   with the same estimator: **0.2267 px** at 1280x720, dominant period **25.5 px**,
   **0.0089 per period**, with short baseline == long baseline (ratio 0.98), so it is
   white noise and the clip is idle. Simulated floors per cell sit ~2x (nx=80), ~4x
   (nx=31) and ~6x (nx=56) below it.

3. **Signal is not noise-invariant**, so gain is only meaningful with a stated model.
   Retention `gain(σ)/gain(clean)` at +0.10 mm, x axis: nx=80 dm1 **99%** at σ=4 and
   **84%** at σ=16, against nx=31 dm1 **72%** and **59%**.

**What the calibrated model supports:** at the render scale, `radius_mm=0.5`,
`dis_mode=1`, nx=80 has the highest clean response on both axes (x 9.10 vs 4.74 px/mm;
y 11.69 vs 10.90) and the best retention, so the shipped 80x80 is competitive or better
on this evidence. The process scale and the `radius_mm=1.5` block have not been
re-ranked.

So: report and compare `noise_models` (gain, floor and `response_retention` per σ, all
measured under one model), not `gain / floor_px`. Every scale records
`null.floor_basis: render_seed_reproducibility` so the distinction is machine-readable.

## The quantities that are still worth watching

- The marker grid is authored **across the camera image**, so a cell is
  `render_width / nx`. It is not a millimetre quantity: a millimetre cell times the
  *probe-local* px/mm understates it by about half, because the probe-local scale
  (~13.2 px/mm) differs from the camera-grid scale (~27 px/mm). `cell_px` is the
  camera-space value and governs; `nominal_cell_px` is the old millimetre-derived value.
- `effective_patch_px = patch_px * 2**finest_scale` is the correlation window in input
  pixels (16 px on r01, because `dis_mode=1`). `cell_over_effective_patch_px` is a
  useful diagnostic — the ratio hovered around 1 for the patterns that scored well — but
  it is **not** a validated law, since the score was floor-driven.
- **The measured gain is confounded with the ROI size.** The probe patch plus falloff
  taper spans only ~13 px, so DIS regularizes toward the static background; a synthetic
  rigid translation recovers ~17% at r=6.6 px (the ROI) but 81% at r=20. Compare patterns
  only at a fixed radius, ROI and scale.
- **The optimum moves with `radius_mm`** (at r=1.5 the score ordering differs), and
  **`overall.score` is the unweighted mean of the two scales**. Read `scale_scores` and
  `scale_ranks` before trusting a winner.

## Top-level fields

| Field | Meaning |
|---|---|
| `texture` | The marker pattern actually measured: PNG path + sha256, `grid` `[nx, ny]`, and the **physical** cell `cell_mm = [vector_width_mm/nx, vector_height_mm/ny]`. `grid`/`cell_mm` are `null` rather than assumed when the pattern cannot be established. |
| `probe` | `group`, `origin`, `basis`, `radius_mm`, `falloff`, and `affected_vertices` (how many vertices outside the probe group actually move). |
| `tracker` | `dis_preset`, `dis_mode`, `patch_px` (the nominal patch) and `finest_scale` (the pyramid level `dis_mode` selects, which is what turns `patch_px` into `effective_patch_px`). |
| `overall` | `score` (mean over scales and axes), `verdict` (worst axis wins), `best_scale`, `best_axis`. |
| `warnings` | The honesty channel. Read it before the score. |
| `notes` | Measured context, including which pattern source was used and the ROI-size confound. |

## Per-scale fields

| Field | Meaning |
|---|---|
| `px_per_mm` | The scale's **probe-local** scalar: `render_manifest.probe_px_per_mm.p50`, halved at the process scale (13.248 at render, 6.624 at process on r01). Used for the sub-pixel warnings and the marker cell in pixels. **Never** a gain divisor — the per-axis `scale_px_per_mm` is, and the two differ (the probe-local p50 is 13.248 while the mesh-wide p50 is 13.430). |
| `mesh_px_per_mm` | The **mesh-wide** p50 (13.430 at render): p50 over all 7341 mesh edges of projected px / 3D mm. Recorded for context only; never used as a gain divisor or as the probe scale. |
| `displacement_px_per_step` | `step_mm * px_per_mm`: how many pixels a single 0.01 mm step is at that scale. |
| `cell_px` | The **camera-space** marker cell, `width / nx`. The pattern grid is defined across the camera image, so this is the cell the estimator sees. `cell_px_source` records that. |
| `nominal_cell_px` | The old millimetre-derived cell (`cell_mm[0] * px_per_mm`) at the probe-local scale; about half of `cell_px`, kept only so an older expectation can be compared. Not the governing value. |
| `effective_patch_px` | `patch_px * 2**finest_scale`, the correlation window in **input** pixels. 16 px on r01, because `dis_mode=1`. |
| `cell_over_patch_px` | `cell_px / patch_px`, the nominal ratio. |
| `cell_over_effective_patch_px` | `cell_px / effective_patch_px` — **the ratio to steer by.** Near 1 is where the measured optima sit. |
| `null_floor_px` | The largest per-axis estimator floor at this scale (see below). |
| `local_scale_source` | `per_axis` (measured per-axis scales), `p50_fallback` (old manifest without per-axis keys), or `unavailable`. |
| `score`, `verdict` | Mean/aggregate of the axes. |

## Per-axis fields

```json
{"axis": "x", "direction_px": [-1.0, 0.000589], "direction_source": "centroid_regression",
 "points": [{"step_mm": 0.01, "applied_px": 0.1325, "response_px": 0.081, "above_floor": true}],
 "summary": {"gain_px_per_mm": 9.247, "gain_mm_per_mm": 0.698, "scale_px_per_mm": 13.248,
             "gain_gate": "derived", "snr": 33.5, "spearman": 1.0,
             "resolvable_step_mm": 0.01, "saturation_mm": null,
             "cross_axis_leak": 0.038, "score": 32.25, "verdict": "resolvable",
             "floor_px": 0.0276, "floor_source": "null_projection_p95",
             "px_per_mm_axis": 13.248}}
```

- `direction_px` — the unit image-space direction the probe moves for this axis,
  measured from `render_manifest.frames[].centroid_px` by regressing the projected
  centroid on `step_mm` (`direction_source: "centroid_regression"`). Fallbacks are
  a two-point finite difference and, when no centroids exist at all, the ROI-mean
  flow at the largest step. The per-frame `roi.u/v` is deliberately not used: the
  ROI is fixed at the reference centroid.
- `points[].applied_px` — the applied displacement **in pixels for that axis**
  (`step_mm * px_per_mm_axis`), not `step_mm * px_per_mm`. The isotropic scalar
  would overstate z (1.519 px/mm) by ~8.7x.
- `floor_px` — p95 of the null-frame flow projected on `direction_px`
  (`floor_source: null_projection_p95`). If the null array is missing the report
  falls back to the flow report's scalar magnitude p95 and says so
  (`reported_magnitude_p95_fallback`), which is a different quantity.
- `gain_px_per_mm` — **the primary sensitivity**: the OLS slope of `response_px`
  against `applied_mm` over the steps whose `|response| > floor_px` (at least
  three such points, else `null`). Units: measured pixels of probe response per
  millimetre applied. This is the number to rank patterns by.
- `scale_px_per_mm` — the probe-local, per-axis scale from
  `render_manifest.probe_px_per_mm[axis]`, rescaled to this scale's pixel grid.
- `gain_mm_per_mm` — derived as `gain_px_per_mm / scale_px_per_mm` **only** when
  `scale_px_per_mm >= min_local_scale_px_per_mm` (currently **1.0**). Otherwise it
  is `null` with `gain_gate` = `below_min_local_scale` (or `scale_unknown`), plus
  a warning. Dividing by a near-zero scale would manufacture a huge ratio no
  camera could observe; the gate makes that structurally impossible. The current
  process-scale z axis (0.760 px/mm) is gated for exactly this reason.
- `snr` — `|gain_px_per_mm| * max_mm / max(floor_px, eps)`. Both numerator and
  denominator are in pixels, so it is dimensionally consistent and comparable.
- `spearman` — rank correlation of `response_px` against `applied_mm` over the
  axis's steps, clamped to `[0, 1]`. A 1.0 means the response grows monotonically.
- `cross_axis_leak` — mean perpendicular response over mean in-axis response,
  clamped to `[0, 1]`.
- `resolvable_step_mm` — the smallest step whose `|response| > floor_px`, else
  `null`. This is the "is a single step measurable" answer, and it is a floor
  comparison, not a pixel-size comparison.
- `saturation_mm` — the first step where the response stops increasing (a decrease
  after a rise), else `null`.
- `px_per_mm_axis` — the probe-local px/mm **measured independently** from the
  centroid regression at this scale. It should agree with `scale_px_per_mm`; a
  >25 % disagreement is reported as a note.
- `score` — `snr * spearman * (1 - min(leak, 0.5))`. A dimensionless ranking
  number, not a percentage.
- `verdict` — in precedence order:
  - `ambiguous` — the tracker's `reference_texture_known_answer` failed, i.e. a
    known shift applied to the *actual rendered marker* was not recovered. A
    periodic pattern aliases a sub-period shift onto its period, so direction and
    magnitude are untrustworthy and no score may promote the pattern. This
    outranks everything else.
  - `resolvable` — `gain_px_per_mm` is not `null` and `snr >= 3`.
  - `marginal` — `gain_px_per_mm` is not `null` and `snr < 3`.
  - `unresolvable` — `gain_px_per_mm` is `null`.

`overall.verdict` is `resolvable` only if every scale and axis is; `ambiguous` if
any scale is; otherwise the worst remaining verdict.

## Honest limits (measured, not assumed)

1. **Sub-pixel steps, and why they still measure.** At 500x500 the probe scale is
   ~13.25 px/mm, so 0.01 mm is ~0.13 px and the whole 0.1 mm sweep is ~1.33 px.
   The measured zero-input floor is ~0.03 px, so a step is ~4.5x above the floor
   and the ROI mean genuinely resolves single steps at the render scale
   (`resolvable_step_mm = 0.01`). **"Below one pixel" and "below the estimator
   floor" are different statements**; the report states the step and the floor
   separately in pixels. What is lost below one pixel is per-pixel precision, not
   the ROI-averaged response. At the process scale the same step is ~0.066 px and
   the floor comparison decides, not the pixel size.
2. **The measured gain is confounded with the ROI size.** The probe patch plus its
   falloff taper spans only ~13 px (1-2 DIS patches of 8 px), so DIS regularises
   toward the static background. Injecting a synthetic rigid translation of the
   same 1.3248 px recovers ~17 % at a patch radius of 6.6 px (equal to the ROI),
   51 % at r 10, 81 % at r 20 and 95 % at r 40. Relative ranking of marker patterns
   is valid **only** while `radius_mm`, the ROI and the scale are held fixed. That
   is why `run_optimise` varies `nx`/`ny`/`seed` and never the radius.
3. **The render-scale deficit is measurement-region resolution**, not noise,
   aliasing or a diluted average: the real flow's radial profile is flat inside the
   ROI (-0.9162, -0.9096, -0.8725 px at r 0-2, 2-4, 4-6.6), the texture-level
   known-answer passed (2 px recovered as 2.0000 px), and the coherent motion is
   measured at about 0.67x of the applied displacement. Only synthetic *periodic*
   textures alias, and that is what the `ambiguous` verdict catches.
4. **The process scale is much weaker.** At 250x250 the region is ~6.5 px, under
   one DIS patch, and the response collapses while the forward-backward error
   grows. A low process score is a real result about the runtime resolution, not a
   bug in the scorer.

## The three pre-registered hypotheses, as the artifact shows them

The report deliberately records the numbers needed to test these rather than
asserting them:

- **(a) the cell should be comparable to the DIS patch.** `cell_px` and
  `cell_over_patch_px` are per scale. For the shipped nx=80 pattern at 500x500 the
  cell is 3.06 px = 0.38 of the 8 px patch; at 250x250 it is 1.53 px, below
  Nyquist, and the report warns. Measured outcome: the hypothesis is
  **scale-dependent, not a straight win** — nx=80 is the *best* pattern at the render
  scale (score 36.77) and near the worst at the process scale (1.77), so no single
  nx is best on both. That is why the report ranks per scale.
- **(b) an 80x80 grid cannot track at 250x250.** Supported at the process scale: that
  is the `cell_px < 2` warning plus the measured collapse of the process responses
  (nx=80 ranks 10/10 there).
- **(c) no single 0.01 mm step is resolvable.** Falsified at the render scale by
  measurement: `resolvable_step_mm` is 0.01 on x/y (and 0.04 on z) because the ROI
  aggregation clears the ~0.03 px floor; it is not assumed either way.

## Warnings you should expect

- `step_mm=... is ... px at <scale>, i.e. below one pixel` — a per-pixel precision
  note that also states the per-axis `floor_px`, so you can see whether the step
  clears the floor.
- `marker cell is ... px at <scale> ... below 2 px` — the cell is at or under the
  DIS patch Nyquist limit.
- `no step is resolvable on axis ... at <scale>` — every `|response|` is inside
  the estimator floor.
- `the null floor (...) exceeds the full ... mm response` — the aggregate response
  never rises out of the floor.
- `axis ... has a probe-local scale of ... below min_local_scale_px_per_mm` — the
  derived `gain_mm_per_mm` was refused.
- `reference_texture_known_answer FAILED ...` — the scale's axes are `ambiguous`;
  the warning quotes measured vs expected shift.
- `the probe ROI is not fully inside the ... frame` — the ROI was truncated at
  that scale and only the pixels actually inside were scored.

## Using the optimizer

```
orisys-design optimise <revision_dir> [--output-dir DIR] [--grid LIST]
                       [--seeds LIST] [--search-resolution N] [--top N] [--dry-run]
```

- The default grid is `[15, 20, 24, 28, 31, 36, 42, 48, 56, 64, 80]` with seed `7`.
  It **brackets the DIS-patch-matched cell** (an 8 px patch is ~0.6 mm, i.e. nx ~ 28-36
  for this marker page), and it includes nx=80 because that is the opposite case, not
  because it aliases: measured on the real run nx=80 **wins the render scale**
  (36.77 at 500x500) and **loses the process scale** (1.77 at 250x250, rank 10/10),
  because its cell is 3.06 px at 500x500 but 1.53 px at 250x250. The shipped random
  texture does **not** alias (a known 2 px shift is recovered as 2.0000 px); aliasing
  onto the DIS period was measured only for strongly periodic synthetic textures, which
  is why the report carries an `ambiguous` verdict gate rather than a score penalty.
- Candidates are `(nx, ny, seed)`, enumerated in a **sorted** order and ranked by
  `(-score, nx, ny, seed)`, so two runs produce byte-identical rankings.
- **The search resolution defaults to `settings.resolution` (the real render scale),
  not to a reduced `250x250`.** A reduced search answers a different question: on the
  real data a 250-based search picks nx=48 (search score 8.94) while nx=80 is the best
  pattern at 500x500 (36.77) and near the worst at 250 (1.77). Use
  `--search-resolution N` only as a deliberate speed/correctness trade, and read the
  **search-vs-confirm gap** note: a large gap is the signal that the search resolution
  was wrong.
- Each candidate regenerates the marker texture, then runs deform -> render ->
  track -> sensitivity at the search resolution. Only the winner is re-run at the full
  `probe.resolution` and its `confirmed_score` recorded.
- `--dry-run` enumerates the candidates and writes the report without rendering
  anything (no candidate directories, no Blender, no cv2).
- The winner is a **measured ranking under a fixed estimator and camera; it is not
  a global optimum**. The report says so in `notes`, and it also warns if the
  top-scoring candidate's verdict is `ambiguous`.
- **`best` is scale-dependent.** `overall.score` is the *unweighted mean* of the
  per-scale scores, which is a modelling choice: on the real data nx=80 wins the render
  scale while a mid-grid pattern balances both scales, so the overall winner need not be
  the best render-scale pattern. Every candidate row therefore carries
  `scale_scores`, `scale_verdicts` and `scale_ranks` (1 = best on that scale), and the
  report carries `ranked_by_scale` alongside the overall `ranked` list and
  `winner.scale_scores`. Read those before trusting the winner.
- Candidate rows carry `score`, `verdict`, `cell_px` (reported at the full render
  scale), `report`, `texture_sha256` and the per-scale fields above. A candidate whose
  run raised is kept in the table with a `null` score, an `error` string, and ranks
  last.

## Reproducing and testing

```
orisys-design sensitivity sensors/<sensor>/<revision>          # score only
orisys-design optimise    sensors/<sensor>/<revision> --dry-run

& "C:\Users\cheun\anaconda3\envs\py310\python.exe" -m pytest tests/test_sensitivity.py tests/test_optimise.py
```

`tests/test_sensitivity.py` includes the primary correctness evidence: a synthetic
flow field with a **known injected gain** (13.248 px/mm) whose OLS recovery is
asserted within 0.2 px/mm (measured recovery 13.2584 px/mm, floor 0.0459 px,
`snr` 28.9), plus the gated-`gain_mm_per_mm` path, the `ambiguous` verdict guard,
resolvable-step selection, the leak penalty, saturation detection, report schema
and determinism. `tests/test_optimise.py` covers enumeration, ranking stability,
winner selection, full-resolution re-confirmation, the dry run and the absence of
any global-optimum claim. Neither test needs Blender or cv2.

### Measured example (this scorer, real `finger-xynova/r01` artifacts)

| scale | axis | `gain_px_per_mm` | `gain_mm_per_mm` | `floor_px` | `resolvable_step_mm` | `snr` | verdict |
|---|---|---|---|---|---|---|---|
| render 500x500, 13.248 px/mm, cell 3.06 px | x | 9.247 | 0.698 | 0.0276 | 0.01 | 33.5 | resolvable |
| | y | 12.019 | 0.680 | 0.0160 | 0.01 | 75.3 | resolvable |
| | z | 0.677 | 0.445 | 0.0160 | 0.04 | 4.24 | resolvable |
| process 250x250, 6.624 px/mm, cell 1.53 px | x | 1.427 | 0.215 | 0.0158 | 0.01 | 9.0 | resolvable |
| | y | 0.931 | 0.105 | 0.0070 | 0.01 | 13.2 | resolvable |
| | z | 0.227 | `null` (gated, scale 0.760 < 1.0) | 0.0070 | 0.03 | 3.23 | resolvable |

`overall.score` 22.22, `overall.verdict` `resolvable`, `best_scale` render,
`best_axis` y. The render-scale `gain_mm_per_mm` of ~0.68-0.70 is the ~0.67x
coherent-motion shrinkage described above, not a noise floor. These numbers are
re-produced by re-running `sensitivity` after any change to the upstream stages.