Handling Angled-Camera Parallax in Position Scoring

This walkthrough sits under Position Validation Algorithms for Planograms and fixes one precise failure the parent stage names but does not fully solve: systematic position error at the frame periphery when a tall or deep fixture is shot from a low or off-center mount. The parent stage projects pixels onto a metric shelf grid with a single planar homography, and that assumption holds beautifully in the middle of the frame and breaks at the edges. A single plane can only be tangent to the fixture at one depth; every facing that sits nearer or farther than that depth is displaced by parallax, and the displacement grows radially outward from the principal point. The tell is unmistakable — the center of the shelf scores clean while the top and bottom corners drift the same direction across every capture, so slots near the frame edge read shifted or vacant for a reason that has nothing to do with merchandising. This page diagnoses that bias, replaces the single plane with a piecewise homography fitted per shelf tier, applies a depth-aware correction where a depth sensor is available, and adds a periphery-aware guard so edge slots earn a wider band instead of a false failure. Each step is independently verifiable against the metric residuals the parent stage already emits.

Prerequisites & Context Jump to heading#

This page runs after the parent projection stage and consumes its outputs; it does not re-detect anything. Confirm the following are in place before applying it:

• Runtime: Python 3.11+ with numpy and opencv-python on the scoring host. No GPU is required — the homography fits and residual checks are cheap CPU work at fixture scope.
• Metric residuals from the parent stage: each detection must already carry its projected centroid in fixture millimetre coordinates and its signed offset from the assigned slot anchor, the output of the single-homography pass in Position Validation Algorithms for Planograms. You also need the pixel position of each centroid so residuals can be binned by distance from the principal point — that radial binning is the whole diagnosis.
• Known mount geometry: the camera’s optical-center height above the fixture base and its lateral offset from fixture center, in millimetres. Parallax scales with object depth over mount distance, so the depth-aware correction in Step 3 is only as good as this geometry.
• Per-tier fiducials: at least four coplanar reference markers on each shelf tier — shelf-edge dividers, lip decals, or printed targets — expressed in the fixture millimetre frame. A single fixture-wide set of markers cannot fit a per-tier plane; each tier needs its own markers at its own depth.
• A slot-to-tier map: every slot_id tagged with the tier_id it belongs to, so a projected centroid is routed through the correct tier homography.

A note on terms: parallax here is the apparent shift of a facing relative to the fixture plane caused by its depth differing from the plane the homography was fitted to. It is a geometric error, not a detection error — no amount of retuning iou_floor or the assignment gate will remove it, because the input coordinates are already wrong before the assignment runs.

Step 1 — Diagnose Parallax: Center Clean, Edges Biased Radially Jump to heading#

Before changing the projection, prove the failure is parallax and not calibration drift or a bumped mount. The signature is specific: on a single planar fit the residual is near zero at the principal point and grows monotonically with radial distance from it, and the direction is consistent across captures. Drift, by contrast, biases the whole frame uniformly; a bump biases it and then holds steady. Bin the metric residuals by pixel radius from the principal point and read the profile.

from __future__ import annotations
from dataclasses import dataclass
import numpy as np


@dataclass(frozen=True)
class ProjectedDetection:
    slot_id: str
    tier_id: str
    residual_mm: np.ndarray   # (2,) signed offset from slot anchor, fixture frame
    frame_xy_px: np.ndarray   # (2,) pixel centroid, for radial binning


def radial_bias_profile(
    dets: list[ProjectedDetection],
    principal_point_px: tuple[float, float],
    n_bins: int = 4,
) -> list[tuple[float, float]]:
    """Mean residual magnitude as a function of distance from the image center.

    A clean planar fit returns a roughly flat profile; parallax on a single
    homography returns a profile that climbs with radius. That monotonic climb,
    with a near-zero first bin, is the diagnostic signature this page targets.
    """
    if not dets:
        raise ValueError("no detections to profile")
    cx, cy = principal_point_px
    radii = np.array([np.hypot(d.frame_xy_px[0] - cx, d.frame_xy_px[1] - cy) for d in dets])
    mag = np.array([float(np.linalg.norm(d.residual_mm)) for d in dets])
    edges = np.linspace(0.0, float(radii.max()) + 1e-6, n_bins + 1)
    profile: list[tuple[float, float]] = []
    for lo, hi in zip(edges[:-1], edges[1:]):
        sel = (radii >= lo) & (radii < hi)
        profile.append((float((lo + hi) / 2.0), float(mag[sel].mean()) if sel.any() else 0.0))
    return profile

If the innermost bin sits under roughly 4 mm while the outermost climbs past 20 mm, you have parallax and should proceed. If every bin is elevated by the same amount, stop here — that is drift, and it belongs in Recovering from Homography Calibration Failures, not in a piecewise re-fit.

Step 2 — Switch to a Piecewise Homography Per Shelf Tier Jump to heading#

A single plane is tangent to the fixture at one depth. A tall gondola or a deep endcap presents several depths — each shelf lip sits farther forward than the one above it — so no single plane fits all of them. The fix is to fit one homography per tier from that tier’s own fiducials, so each plane is locally tangent where a global plane is not. Every tier carries its own reprojection residual, and a tier whose fit exceeds the guard fails loud rather than quietly projecting garbage into the assignment.

import cv2


@dataclass(frozen=True)
class TierHomography:
    tier_id: str
    H: np.ndarray            # (3,3) pixel -> fixture mm for this tier's plane
    rms_mm: float


def fit_tier_homographies(
    tier_fiducials: dict[str, tuple[np.ndarray, np.ndarray]],
    max_rms_mm: float = 6.0,
    ransac_thresh_px: float = 3.0,
) -> dict[str, TierHomography]:
    """Fit one homography per shelf tier from that tier's own fiducials.

    Each tier plane is fitted to markers at its own depth, so the planar
    assumption holds locally where a single fixture-wide plane breaks. Raises
    when a tier has too few markers or its residual exceeds the guard, so a
    bad tier is skipped rather than corrupting the whole fixture's scores.
    """
    fitted: dict[str, TierHomography] = {}
    for tier_id, (img_pts, mm_pts) in tier_fiducials.items():
        if len(img_pts) < 4 or len(img_pts) != len(mm_pts):
            raise ValueError(f"tier {tier_id}: need >= 4 matched fiducials")
        H, mask = cv2.findHomography(
            img_pts, mm_pts, method=cv2.RANSAC, ransacReprojThreshold=ransac_thresh_px
        )
        if H is None or int(mask.sum()) < 4:
            raise ValueError(f"tier {tier_id}: degenerate homography")
        proj = cv2.perspectiveTransform(img_pts.reshape(-1, 1, 2), H).reshape(-1, 2)
        rms = float(np.sqrt(np.mean(np.sum((proj - mm_pts) ** 2, axis=1))))
        if rms > max_rms_mm:
            raise ValueError(f"tier {tier_id}: rms {rms:.1f}mm exceeds {max_rms_mm}mm guard")
        fitted[tier_id] = TierHomography(tier_id, H, rms)
    return fitted


def project_in_tier(pt_px: np.ndarray, tier: TierHomography) -> np.ndarray:
    """Project one pixel centroid through its own tier's homography."""
    return cv2.perspectiveTransform(pt_px.reshape(-1, 1, 2), tier.H).reshape(2)

Route each detection to fitted[det.tier_id] using the slot-to-tier map, then re-run the parent stage’s assignment on the corrected coordinates. Nothing downstream changes — the assignment, the slot_iou, and the graded state all consume the same metric frame; they simply receive coordinates that are now correct at the periphery. The IoU floor those states depend on is tuned separately in Calibrating IoU Thresholds for Dense Endcap Displays, and a piecewise fit is what makes those thresholds meaningful on a deep display.

Step 3 — Apply Depth-Aware Correction Where Depth Exists Jump to heading#

Piecewise projection removes the depth between tiers; a residual parallax remains when a facing sits proud of its own tier plane — a bottle standing forward of the shelf lip, a boxed SKU pushed to the front rail. Where a stereo rig or LiDAR supplies a per-facing depth, subtract the remaining parallax directly. The shift is the object’s depth beyond the tier plane, scaled by depth over mount distance, applied along the lever from the mount’s principal ray.

@dataclass(frozen=True)
class MountGeometry:
    height_mm: float             # optical center height above the fixture base
    horizontal_offset_mm: float  # lateral offset of the mount from fixture center


def depth_correct(
    centroid_mm: np.ndarray,
    slot_depth_mm: float,        # measured forward position of the facing
    tier_plane_depth_mm: float,  # depth the tier homography was fitted to
    mount: MountGeometry,
) -> np.ndarray:
    """Remove residual parallax for a facing that sits proud of its tier plane.

    Parallax scales with (object_depth_beyond_plane / mount_distance). A facing
    standing proud of the tier plane projects outward along the ray from the
    mount, so subtract that component when a depth sensor supplies slot_depth_mm.
    Returns the centroid unchanged when the facing sits on the plane.
    """
    delta_depth = slot_depth_mm - tier_plane_depth_mm
    if delta_depth == 0.0:
        return centroid_mm
    mount_distance = max(mount.height_mm, 1.0)
    scale = delta_depth / mount_distance
    lever = centroid_mm - np.array([mount.horizontal_offset_mm, 0.0])
    return centroid_mm - lever * scale

Depth-aware correction is optional and strictly additive: without a depth sensor, skip it and rely on Steps 2 and 4. Never fabricate a slot_depth_mm from a default — a wrong depth introduces a correction worse than the parallax it was meant to remove.

Step 4 — Add a Periphery-Aware Offset Guard Jump to heading#

Even after a piecewise fit, the last few millimetres of residual are unavoidable at the extreme corners, where marker coverage is thinnest and the plane approximation is weakest. Rather than let those slots flip to shifted, widen the in-position band as a function of radial distance from the principal point. Center slots keep the base band; corner slots earn a proportionally wider one, so a corner facing that is genuinely in place is not punished for the residual the geometry could not remove.

def peripheral_band_mm(
    base_band_mm: float,
    frame_xy_px: np.ndarray,
    principal_point_px: tuple[float, float],
    frame_radius_px: float,
    max_widen: float = 1.8,
) -> float:
    """Widen the in-position band for slots far from the image center.

    Residual parallax that piecewise projection cannot fully remove grows with
    radius, so an edge slot earns up to `max_widen` times the base band while a
    center slot keeps it unchanged. Keeps a true corner facing from reading as a
    false 'shifted' without loosening the band where the fit is already clean.
    """
    if frame_radius_px <= 0:
        raise ValueError("frame_radius_px must be positive")
    cx, cy = principal_point_px
    r = float(np.hypot(frame_xy_px[0] - cx, frame_xy_px[1] - cy))
    frac = min(r / frame_radius_px, 1.0)
    return round(base_band_mm * (1.0 + (max_widen - 1.0) * frac), 2)

Keep max_widen conservative — a value of 1.8 turns a 12 mm base band into at most 21.6 mm at the extreme corner. Widen further only if the audit reconciliation shows edge false-positives persisting; widening too aggressively hides genuine misplacements in exactly the zone where parallax already made you nervous.

Verification & Testing Jump to heading#

Confirm each correction deterministically against the residuals, not by eyeballing a heatmap:

1. Diagnosis fires on the right shape. Feed radial_bias_profile a synthetic set whose residual magnitude grows with radius and assert the profile is monotonically increasing with a first bin under 4 mm; feed a uniformly biased set and assert the profile is flat, so the diagnosis does not misread drift as parallax.
2. Edge bias is removed after the piecewise fit. Score a tall fixture through the single plane, then through fit_tier_homographies, and assert the outermost radial bin drops from its single-plane value (e.g. past 20 mm) to within the per-tier rms_mm guard of 6 mm.
3. The center is unchanged. Assert the innermost radial bin’s mean residual moves by less than 1 mm between the single-plane and piecewise runs — the fix must not disturb the slots that were already correct.
4. Depth correction is a no-op on the plane. Call depth_correct with slot_depth_mm == tier_plane_depth_mm and assert the centroid is returned unchanged, so a facing sitting on its tier plane is never shifted.
5. The guard widens with radius, monotonically. Assert peripheral_band_mm returns the base band at the principal point and a strictly larger value at the frame corner, and that it never exceeds base_band_mm * max_widen.
6. A degenerate tier fails loud. Hand fit_tier_homographies a tier with three markers or a collinear set and assert it raises, so a bad tier is skipped rather than silently projecting the whole fixture wrong.

A healthy run shows the in_position share on the top and bottom tiers rising to match the center tier’s, with shifted records no longer clustered in one radial direction.

Troubleshooting Jump to heading#

Related Jump to heading#

• Position Validation Algorithms for Planograms — the parent stage whose single-homography projection this page corrects at the periphery
• Calibrating IoU Thresholds for Dense Endcap Displays — the sibling that tunes the overlap thresholds a piecewise fit makes trustworthy on deep displays
• Recovering from Homography Calibration Failures — the recovery path when the fit itself has degraded rather than merely broken at the edges