orisys-sensor-design/docs/probe-method.md

Reading the probe sensitivity report

This is the operator's guide to sensitivity_report.json and optimizer_report.json. The interface itself is frozen in probe-pipeline.md; this document explains what the numbers mean, what they do not mean, and how to use the optimizer.

The bench deforms the sensing surface through a known positive x/y/z sweep (0.01 mm .. 0.10 mm in 0.01 mm steps) in a local probe-average basis, renders what cam_sensor sees, and measures DIS optical flow against the zero-displacement reference at two scales (the real render and the runtime's processed resolution).

deform -> render -> track (DIS flow) -> sensitivity (score + report) -> optimise

Every stage writes JSON under build/probe/<sensor>/<revision>/. The sensitivity stage reads render_manifest.json, flow_report.json and flow/<scale>/*.npy and writes sensitivity_report.json. It never re-runs a stage; use the CLI's --skip-deform/--skip-render/--skip-track to reuse existing artifacts.

The one thing to remember

Sensitivity here is a relative ranking under a fixed estimator (DIS) and a fixed camera. It is not force calibration. The absolute gain_mm_per_mm carries the measurement geometry with it, so only compare scores produced with the same radius_mm, ROI, resolution and tracker settings.

The floor is not a noise model (read this first)

An earlier version of this document stated that the marker cell must match the estimator's effective window, and that nx=31 was the best pattern. That is retracted. The conclusion was driven by floor_px, and floor_px is not a valid noise model.

floor_px is measured as reference vs a re-render at seed + 1 — Cycles antialiasing jitter, ~5 grey levels RMS on half the pixels, scaling with edge contrast. It cannot rank patterns, for three measured reasons:

1. Its trend reverses under additive noise. p95 |flow| over the ROI, same DIS config:

   pattern	render-seed	injected σ=2	σ=4
   nx=80	0.0292	0.0161	0.0278
   nx=31	0.0376	0.0084	0.0182
   nx=56	0.0134	0.0102	0.0203

2. It understates the real floor, coarse patterns most. Measured on idle dd02 footage with the same estimator: 0.2267 px at 1280x720, dominant period 25.5 px, 0.0089 per period, with short baseline == long baseline (ratio 0.98), so it is white noise and the clip is idle. Simulated floors per cell sit ~2x (nx=80), ~4x (nx=31) and ~6x (nx=56) below it.

3. Signal is not noise-invariant, so gain is only meaningful with a stated model. Retention gain(σ)/gain(clean) at +0.10 mm, x axis: nx=80 dm1 99% at σ=4 and 84% at σ=16, against nx=31 dm1 72% and 59%.

What the calibrated model supports: at the render scale, radius_mm=0.5, dis_mode=1, nx=80 has the highest clean response on both axes (x 9.10 vs 4.74 px/mm; y 11.69 vs 10.90) and the best retention, so the shipped 80x80 is competitive or better on this evidence. The process scale and the radius_mm=1.5 block have not been re-ranked.

So: report and compare noise_models (gain, floor and response_retention per σ, all measured under one model), not gain / floor_px. Every scale records null.floor_basis: render_seed_reproducibility so the distinction is machine-readable.

The quantities that are still worth watching

• The marker grid is authored across the camera image, so a cell is render_width / nx. It is not a millimetre quantity: a millimetre cell times the probe-local px/mm understates it by about half, because the probe-local scale (~13.2 px/mm) differs from the camera-grid scale (~27 px/mm). cell_px is the camera-space value and governs; nominal_cell_px is the old millimetre-derived value.
• effective_patch_px = patch_px * 2**finest_scale is the correlation window in input pixels (16 px on r01, because dis_mode=1). cell_over_effective_patch_px is a useful diagnostic — the ratio hovered around 1 for the patterns that scored well — but it is not a validated law, since the score was floor-driven.
• The measured gain is confounded with the ROI size. The probe patch plus falloff taper spans only ~13 px, so DIS regularizes toward the static background; a synthetic rigid translation recovers ~17% at r=6.6 px (the ROI) but 81% at r=20. Compare patterns only at a fixed radius, ROI and scale.
• The optimum moves with radius_mm (at r=1.5 the score ordering differs), and overall.score is the unweighted mean of the two scales. Read scale_scores and scale_ranks before trusting a winner.

Top-level fields

Field	Meaning
texture	The marker pattern actually measured: PNG path + sha256, grid [nx, ny], and the physical cell cell_mm = [vector_width_mm/nx, vector_height_mm/ny]. grid/cell_mm are null rather than assumed when the pattern cannot be established.
probe	group, origin, basis, radius_mm, falloff, and affected_vertices (how many vertices outside the probe group actually move).
tracker	dis_preset, dis_mode, patch_px (the nominal patch) and finest_scale (the pyramid level dis_mode selects, which is what turns patch_px into effective_patch_px).
overall	score (mean over scales and axes), verdict (worst axis wins), best_scale, best_axis.
warnings	The honesty channel. Read it before the score.
notes	Measured context, including which pattern source was used and the ROI-size confound.

Per-scale fields

Field	Meaning
px_per_mm	The scale's probe-local scalar: render_manifest.probe_px_per_mm.p50, halved at the process scale (13.248 at render, 6.624 at process on r01). Used for the sub-pixel warnings and the marker cell in pixels. Never a gain divisor — the per-axis scale_px_per_mm is, and the two differ (the probe-local p50 is 13.248 while the mesh-wide p50 is 13.430).
mesh_px_per_mm	The mesh-wide p50 (13.430 at render): p50 over all 7341 mesh edges of projected px / 3D mm. Recorded for context only; never used as a gain divisor or as the probe scale.
displacement_px_per_step	step_mm * px_per_mm: how many pixels a single 0.01 mm step is at that scale.
cell_px	The camera-space marker cell, width / nx. The pattern grid is defined across the camera image, so this is the cell the estimator sees. cell_px_source records that.
nominal_cell_px	The old millimetre-derived cell (cell_mm[0] * px_per_mm) at the probe-local scale; about half of cell_px, kept only so an older expectation can be compared. Not the governing value.
effective_patch_px	patch_px * 2**finest_scale, the correlation window in input pixels. 16 px on r01, because dis_mode=1.
cell_over_patch_px	cell_px / patch_px, the nominal ratio.
cell_over_effective_patch_px	cell_px / effective_patch_px — the ratio to steer by. Near 1 is where the measured optima sit.
null_floor_px	The largest per-axis estimator floor at this scale (see below).
local_scale_source	per_axis (measured per-axis scales), p50_fallback (old manifest without per-axis keys), or unavailable.
score, verdict	Mean/aggregate of the axes.

Per-axis fields

{"axis": "x", "direction_px": [-1.0, 0.000589], "direction_source": "centroid_regression",
 "points": [{"step_mm": 0.01, "applied_px": 0.1325, "response_px": 0.081, "above_floor": true}],
 "summary": {"gain_px_per_mm": 9.247, "gain_mm_per_mm": 0.698, "scale_px_per_mm": 13.248,
             "gain_gate": "derived", "snr": 33.5, "spearman": 1.0,
             "resolvable_step_mm": 0.01, "saturation_mm": null,
             "cross_axis_leak": 0.038, "score": 32.25, "verdict": "resolvable",
             "floor_px": 0.0276, "floor_source": "null_projection_p95",
             "px_per_mm_axis": 13.248}}

• direction_px — the unit image-space direction the probe moves for this axis, measured from render_manifest.frames[].centroid_px by regressing the projected centroid on step_mm (direction_source: "centroid_regression"). Fallbacks are a two-point finite difference and, when no centroids exist at all, the ROI-mean flow at the largest step. The per-frame roi.u/v is deliberately not used: the ROI is fixed at the reference centroid.
• points[].applied_px — the applied displacement in pixels for that axis (step_mm * px_per_mm_axis), not step_mm * px_per_mm. The isotropic scalar would overstate z (1.519 px/mm) by ~8.7x.
• floor_px — p95 of the null-frame flow projected on direction_px (floor_source: null_projection_p95). If the null array is missing the report falls back to the flow report's scalar magnitude p95 and says so (reported_magnitude_p95_fallback), which is a different quantity.
• gain_px_per_mm — the primary sensitivity: the OLS slope of response_px against applied_mm over the steps whose |response| > floor_px (at least three such points, else null). Units: measured pixels of probe response per millimetre applied. This is the number to rank patterns by.
• scale_px_per_mm — the probe-local, per-axis scale from render_manifest.probe_px_per_mm[axis], rescaled to this scale's pixel grid.
• gain_mm_per_mm — derived as gain_px_per_mm / scale_px_per_mm only when scale_px_per_mm >= min_local_scale_px_per_mm (currently 1.0). Otherwise it is null with gain_gate = below_min_local_scale (or scale_unknown), plus a warning. Dividing by a near-zero scale would manufacture a huge ratio no camera could observe; the gate makes that structurally impossible. The current process-scale z axis (0.760 px/mm) is gated for exactly this reason.
• snr — |gain_px_per_mm| * max_mm / max(floor_px, eps). Both numerator and denominator are in pixels, so it is dimensionally consistent and comparable.
• spearman — rank correlation of response_px against applied_mm over the axis's steps, clamped to [0, 1]. A 1.0 means the response grows monotonically.
• cross_axis_leak — mean perpendicular response over mean in-axis response, clamped to [0, 1].
• resolvable_step_mm — the smallest step whose |response| > floor_px, else null. This is the "is a single step measurable" answer, and it is a floor comparison, not a pixel-size comparison.
• saturation_mm — the first step where the response stops increasing (a decrease after a rise), else null.
• px_per_mm_axis — the probe-local px/mm measured independently from the centroid regression at this scale. It should agree with scale_px_per_mm; a

  25 % disagreement is reported as a note.

• score — snr * spearman * (1 - min(leak, 0.5)). A dimensionless ranking number, not a percentage.
• verdict — in precedence order:
  • ambiguous — the tracker's reference_texture_known_answer failed, i.e. a known shift applied to the actual rendered marker was not recovered. A periodic pattern aliases a sub-period shift onto its period, so direction and magnitude are untrustworthy and no score may promote the pattern. This outranks everything else.
  • resolvable — gain_px_per_mm is not null and snr >= 3.
  • marginal — gain_px_per_mm is not null and snr < 3.
  • unresolvable — gain_px_per_mm is null.

overall.verdict is resolvable only if every scale and axis is; ambiguous if any scale is; otherwise the worst remaining verdict.

Honest limits (measured, not assumed)

1. Sub-pixel steps, and why they still measure. At 500x500 the probe scale is ~13.25 px/mm, so 0.01 mm is ~0.13 px and the whole 0.1 mm sweep is ~1.33 px. The measured zero-input floor is ~0.03 px, so a step is ~4.5x above the floor and the ROI mean genuinely resolves single steps at the render scale (resolvable_step_mm = 0.01). "Below one pixel" and "below the estimator floor" are different statements; the report states the step and the floor separately in pixels. What is lost below one pixel is per-pixel precision, not the ROI-averaged response. At the process scale the same step is ~0.066 px and the floor comparison decides, not the pixel size.
2. The measured gain is confounded with the ROI size. The probe patch plus its falloff taper spans only ~13 px (1-2 DIS patches of 8 px), so DIS regularises toward the static background. Injecting a synthetic rigid translation of the same 1.3248 px recovers ~17 % at a patch radius of 6.6 px (equal to the ROI), 51 % at r 10, 81 % at r 20 and 95 % at r 40. Relative ranking of marker patterns is valid only while radius_mm, the ROI and the scale are held fixed. That is why run_optimise varies nx/ny/seed and never the radius.
3. The render-scale deficit is measurement-region resolution, not noise, aliasing or a diluted average: the real flow's radial profile is flat inside the ROI (-0.9162, -0.9096, -0.8725 px at r 0-2, 2-4, 4-6.6), the texture-level known-answer passed (2 px recovered as 2.0000 px), and the coherent motion is measured at about 0.67x of the applied displacement. Only synthetic periodic textures alias, and that is what the ambiguous verdict catches.
4. The process scale is much weaker. At 250x250 the region is ~6.5 px, under one DIS patch, and the response collapses while the forward-backward error grows. A low process score is a real result about the runtime resolution, not a bug in the scorer.

The three pre-registered hypotheses, as the artifact shows them

The report deliberately records the numbers needed to test these rather than asserting them:

• (a) the cell should be comparable to the DIS patch. cell_px and cell_over_patch_px are per scale. For the shipped nx=80 pattern at 500x500 the cell is 3.06 px = 0.38 of the 8 px patch; at 250x250 it is 1.53 px, below Nyquist, and the report warns. Measured outcome: the hypothesis is scale-dependent, not a straight win — nx=80 is the best pattern at the render scale (score 36.77) and near the worst at the process scale (1.77), so no single nx is best on both. That is why the report ranks per scale.
• (b) an 80x80 grid cannot track at 250x250. Supported at the process scale: that is the cell_px < 2 warning plus the measured collapse of the process responses (nx=80 ranks 10/10 there).
• (c) no single 0.01 mm step is resolvable. Falsified at the render scale by measurement: resolvable_step_mm is 0.01 on x/y (and 0.04 on z) because the ROI aggregation clears the ~0.03 px floor; it is not assumed either way.

Warnings you should expect

• step_mm=... is ... px at <scale>, i.e. below one pixel — a per-pixel precision note that also states the per-axis floor_px, so you can see whether the step clears the floor.
• marker cell is ... px at <scale> ... below 2 px — the cell is at or under the DIS patch Nyquist limit.
• no step is resolvable on axis ... at <scale> — every |response| is inside the estimator floor.
• the null floor (...) exceeds the full ... mm response — the aggregate response never rises out of the floor.
• axis ... has a probe-local scale of ... below min_local_scale_px_per_mm — the derived gain_mm_per_mm was refused.
• reference_texture_known_answer FAILED ... — the scale's axes are ambiguous; the warning quotes measured vs expected shift.
• the probe ROI is not fully inside the ... frame — the ROI was truncated at that scale and only the pixels actually inside were scored.

Using the optimizer

orisys-design optimise <revision_dir> [--output-dir DIR] [--grid LIST]
                       [--seeds LIST] [--search-resolution N] [--top N] [--dry-run]

• The default grid is [15, 20, 24, 28, 31, 36, 42, 48, 56, 64, 80] with seed 7. It brackets the DIS-patch-matched cell (an 8 px patch is ~0.6 mm, i.e. nx ~ 28-36 for this marker page), and it includes nx=80 because that is the opposite case, not because it aliases: measured on the real run nx=80 wins the render scale (36.77 at 500x500) and loses the process scale (1.77 at 250x250, rank 10/10), because its cell is 3.06 px at 500x500 but 1.53 px at 250x250. The shipped random texture does not alias (a known 2 px shift is recovered as 2.0000 px); aliasing onto the DIS period was measured only for strongly periodic synthetic textures, which is why the report carries an ambiguous verdict gate rather than a score penalty.
• Candidates are (nx, ny, seed), enumerated in a sorted order and ranked by (-score, nx, ny, seed), so two runs produce byte-identical rankings.
• The search resolution defaults to settings.resolution (the real render scale), not to a reduced 250x250. A reduced search answers a different question: on the real data a 250-based search picks nx=48 (search score 8.94) while nx=80 is the best pattern at 500x500 (36.77) and near the worst at 250 (1.77). Use --search-resolution N only as a deliberate speed/correctness trade, and read the search-vs-confirm gap note: a large gap is the signal that the search resolution was wrong.
• Each candidate regenerates the marker texture, then runs deform -> render -> track -> sensitivity at the search resolution. Only the winner is re-run at the full probe.resolution and its confirmed_score recorded.
• --dry-run enumerates the candidates and writes the report without rendering anything (no candidate directories, no Blender, no cv2).
• The winner is a measured ranking under a fixed estimator and camera; it is not a global optimum. The report says so in notes, and it also warns if the top-scoring candidate's verdict is ambiguous.
• best is scale-dependent. overall.score is the unweighted mean of the per-scale scores, which is a modelling choice: on the real data nx=80 wins the render scale while a mid-grid pattern balances both scales, so the overall winner need not be the best render-scale pattern. Every candidate row therefore carries scale_scores, scale_verdicts and scale_ranks (1 = best on that scale), and the report carries ranked_by_scale alongside the overall ranked list and winner.scale_scores. Read those before trusting the winner.
• Candidate rows carry score, verdict, cell_px (reported at the full render scale), report, texture_sha256 and the per-scale fields above. A candidate whose run raised is kept in the table with a null score, an error string, and ranks last.

Reproducing and testing

orisys-design sensitivity sensors/<sensor>/<revision>          # score only
orisys-design optimise    sensors/<sensor>/<revision> --dry-run

& "C:\Users\cheun\anaconda3\envs\py310\python.exe" -m pytest tests/test_sensitivity.py tests/test_optimise.py

tests/test_sensitivity.py includes the primary correctness evidence: a synthetic flow field with a known injected gain (13.248 px/mm) whose OLS recovery is asserted within 0.2 px/mm (measured recovery 13.2584 px/mm, floor 0.0459 px, snr 28.9), plus the gated-gain_mm_per_mm path, the ambiguous verdict guard, resolvable-step selection, the leak penalty, saturation detection, report schema and determinism. tests/test_optimise.py covers enumeration, ranking stability, winner selection, full-resolution re-confirmation, the dry run and the absence of any global-optimum claim. Neither test needs Blender or cv2.

Measured example (this scorer, real finger-xynova/r01 artifacts)

scale	axis	gain_px_per_mm	gain_mm_per_mm	floor_px	resolvable_step_mm	snr	verdict
render 500x500, 13.248 px/mm, cell 3.06 px	x	9.247	0.698	0.0276	0.01	33.5	resolvable
	y	12.019	0.680	0.0160	0.01	75.3	resolvable
	z	0.677	0.445	0.0160	0.04	4.24	resolvable
process 250x250, 6.624 px/mm, cell 1.53 px	x	1.427	0.215	0.0158	0.01	9.0	resolvable
	y	0.931	0.105	0.0070	0.01	13.2	resolvable
	z	0.227	null (gated, scale 0.760 < 1.0)	0.0070	0.03	3.23	resolvable

overall.score 22.22, overall.verdict resolvable, best_scale render, best_axis y. The render-scale gain_mm_per_mm of ~0.68-0.70 is the ~0.67x coherent-motion shrinkage described above, not a noise floor. These numbers are re-produced by re-running sensitivity after any change to the upstream stages.