Recovering from Homography Calibration Failures

This walkthrough sits under Error Handling in Computer Vision Pipelines and solves one precise, recurring failure: a fixed store camera gets bumped by a pallet jack, a fiducial marker is occluded by a returned-goods cart, and the homography that projects pixels onto the fixture-metric grid silently degrades. The transform still fits — it just fits badly, biasing every centroid by a few centimetres in the same direction. If you let that transform through, the downstream tolerance check grades a whole bay shifted in unison and the pipeline emits a confident, wrong compliance score. The homography fit is the recovery counterpart to the position-validation stage: Position Validation Algorithms for Planograms is the stage that computes the projection, and this page is what runs in front of it so a degenerate or degraded fit is caught, replaced with a cached last-good transform, or quarantined for recalibration — never scored against. Each step below is independently verifiable.

Prerequisites & Context Jump to heading#

This stage assumes marker-based projection is already how you remove the camera. It sits between raw detection and tolerance scoring, and it never touches the image itself — it only judges the transform.

• Runtime: Python 3.11+ with numpy and opencv-python on the scoring host; no GPU is required.
• Marker-based projection: each capture carries the pixel coordinates of at least four printed fiducials (ArUco or a fixed corner rig) whose true positions on the fixture-metric grid are known and versioned by registry_revision. The homography is solved from these correspondences.
• A reprojection RMS guard: a single tuning constant — 4.0 px is a sound default for a 1080p fixed mount — above which the fit is treated as degraded rather than usable. This is the same idea as the tolerance floor in the tolerance-band walkthrough, applied one stage earlier.
• A cached last-good homography per fixture_id: the most recent transform that passed every gate, stored with its registry_revision and capture_timestamp so a temporarily blinded camera can keep scoring on the last trustworthy geometry.
• A quarantine sink: the same typed dead-letter path the parent layer already runs, so a held capture keeps its full context and can be replayed after recalibration.

A note on terms: a degenerate fit cannot be solved at all (too few or collinear markers); a degraded fit solves but reprojects poorly (a bumped camera, a partially occluded marker). The recovery path differs for each, which is why the gate reports the reason, not just a boolean.

Step 1 — Validate the Fit Before Trusting It Jump to heading#

A homography is a 3×3 matrix with eight degrees of freedom, so it needs at least four non-collinear point correspondences to solve. But solvability is the low bar. The dangerous case is a fit that solves and still lies: a marker nudged out of position by a bumped mount drags the whole transform, and every projected centroid inherits the bias. Validate three things before the fit is allowed downstream — marker count, reprojection RMS against the guard, and the condition number of the design matrix, which catches near-collinear or clustered markers that produce a numerically unstable solution even when four are present.

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum

import numpy as np


class FitVerdict(str, Enum):
    OK = "ok"                       # passes every gate; safe to score
    DEGENERATE = "degenerate"       # too few / unsolvable markers
    DEGRADED = "degraded"           # solves but reprojects badly


@dataclass(frozen=True, slots=True)
class FitReport:
    verdict: FitVerdict
    marker_count: int
    reprojection_rms_px: float
    condition_number: float
    detail: str


class DegenerateFit(Exception):
    """Raised when a homography cannot or must not be scored against."""
    def __init__(self, report: FitReport) -> None:
        super().__init__(report.detail)
        self.report = report


def validate_fit(
    src_px: np.ndarray,          # detected marker pixels, shape (N, 2)
    dst_mm: np.ndarray,          # known fixture-metric coords, shape (N, 2)
    homography: np.ndarray | None,
    *,
    rms_guard_px: float = 4.0,
    min_markers: int = 4,
    condition_ceiling: float = 1.0e7,
) -> FitReport:
    """Judge a marker-based homography without scoring anything against it."""
    n = int(src_px.shape[0])
    if n < min_markers or homography is None:
        return FitReport(FitVerdict.DEGENERATE, n, float("inf"), float("inf"),
                         f"only {n} usable markers (need {min_markers})")

    cond = float(np.linalg.cond(homography))
    proj = cv2_perspective(src_px, homography)      # (N, 2) in mm
    rms = float(np.sqrt(np.mean(np.sum((proj - dst_mm) ** 2, axis=1))))

    if cond > condition_ceiling:
        return FitReport(FitVerdict.DEGRADED, n, rms, cond,
                         f"ill-conditioned fit (cond={cond:.1e})")
    if rms > rms_guard_px:
        return FitReport(FitVerdict.DEGRADED, n, rms, cond,
                         f"reprojection RMS {rms:.2f}px over {rms_guard_px}px guard")
    return FitReport(FitVerdict.OK, n, rms, cond, "fit within all guards")

The cv2_perspective helper is a thin typed wrapper over cv2.perspectiveTransform; keeping RMS in the metric frame means the guard reads in the same units your markers are surveyed in. The key discipline is that validate_fit reports — it never projects a single detection. Scoring is a separate call that only runs once the verdict is OK.

Step 2 — Fall Back to the Cached Last-Good Homography Jump to heading#

When the fit is DEGENERATE because a marker was momentarily occluded — a shopper, a stacked cart, a glare blowout over one corner — the fixture geometry has not actually changed. The camera is fixed; only this frame is blind. In that case the correct move is not to quarantine but to reuse the last transform that passed every gate for this fixture_id, flag the resulting records as degraded so reporting can down-weight them, and keep scoring. The cache is only trustworthy while the mount is stable, so it carries the registry_revision it was fit under and refuses to serve across a revision bump.

@dataclass(frozen=True, slots=True)
class CachedHomography:
    fixture_id: str
    matrix: np.ndarray
    registry_revision: str
    capture_timestamp: str


class HomographyCache:
    """Last-good transform per fixture, guarded by registry revision."""
    def __init__(self) -> None:
        self._store: dict[str, CachedHomography] = {}

    def put(self, entry: CachedHomography) -> None:
        self._store[entry.fixture_id] = entry

    def get(self, fixture_id: str, registry_revision: str) -> CachedHomography | None:
        entry = self._store.get(fixture_id)
        if entry is None or entry.registry_revision != registry_revision:
            return None                          # stale or absent: cannot fall back
        return entry

Refusing to serve across a registry_revision change is what stops a stale homography from silently outliving a genuine fixture reset. If the planogram was re-laid and the markers re-surveyed, the old transform is wrong even though it once passed — so the cache returns None and the frame is forced down the quarantine path instead, where a fresh calibration is demanded.

Step 3 — Quarantine and Flag for Recalibration Jump to heading#

A DEGRADED verdict means the opposite of an occlusion: the markers were found, the fit solved, and it still reprojects past the guard. That is physical — the camera moved, a marker peeled, or the survey is wrong. There is no trustworthy transform to score against, cached or otherwise, so the fixture is quarantined, a recalibration task is raised for the associate, and no compliance record is emitted. This reuses the parent layer’s dead-letter contract, so the held capture keeps its full FrameContext and can be replayed later.

def recover_homography(
    ctx: "FrameContext",
    report: FitReport,
    homography: np.ndarray | None,
    cache: HomographyCache,
    dlq: "DeadLetterQueue",
) -> tuple[np.ndarray, bool]:
    """Return (usable_homography, is_degraded) or raise to quarantine.

    Never returns a transform that failed the RMS or condition gate.
    """
    if report.verdict is FitVerdict.OK and homography is not None:
        cache.put(CachedHomography(ctx.fixture_id, homography,
                                   ctx.registry_revision, ctx.capture_timestamp))
        return homography, False

    if report.verdict is FitVerdict.DEGENERATE:
        cached = cache.get(ctx.fixture_id, ctx.registry_revision)
        if cached is not None:
            return cached.matrix, True           # score on last-good, flagged
        # no cache under this revision: fall through to quarantine

    dlq.send(ctx, stage="homography", detail=report.detail)
    raise DegenerateFit(report)                  # fixture held, recalibration flagged

The signature is the whole point: the function either hands back a transform that passed every gate (fresh or cached) with an honest degraded flag, or it raises. There is no branch that returns the bad matrix. A degraded fit can never leak into scoring, which is exactly the invariant the parent error-handling layer enforces for every other stage — a frame produces a trustworthy record or a typed dead-letter, never a fabricated verdict.

Step 4 — Replay Quarantined Captures After Recalibration Jump to heading#

Quarantine is not a graveyard. Once an associate re-mounts the camera or re-prints a fiducial and the fixture is recalibrated under a new registry_revision, the held captures should be re-fit against the corrected geometry rather than discarded — the raw shelf photos are still perfectly good, only the transform was wrong. Drain the dead-letter records for that fixture_id, re-run the fit and the gate, and let the outcomes flow: a now-passing fit scores normally, while a capture that still fails (an occlusion that outlived the incident) simply re-quarantines.

def replay_fixture(
    fixture_id: str,
    records: list["FrameContext"],
    fit_fn,                                       # (ctx) -> (src, dst, H)
    cache: HomographyCache,
    dlq: "DeadLetterQueue",
) -> list[tuple[str, FitVerdict]]:
    """Re-fit held captures oldest-first after a recalibration."""
    outcomes: list[tuple[str, FitVerdict]] = []
    for ctx in sorted(records, key=lambda c: c.capture_timestamp):
        src, dst, homography = fit_fn(ctx)
        report = validate_fit(src, dst, homography)
        try:
            _, degraded = recover_homography(ctx, report, homography, cache, dlq)
            outcomes.append((ctx.capture_id, report.verdict))
        except DegenerateFit:
            outcomes.append((ctx.capture_id, FitVerdict.DEGRADED))
    return outcomes

Replaying oldest-first matters: it rebuilds the cache in capture order so a capture_timestamp on a scored record always reflects the geometry that was live when the photo was taken, and a later consumer reconstructing a fixture’s compliance timeline never sees a newer transform applied to an older frame.

Verification & Testing Jump to heading#

Confirm each rule deterministically rather than trusting a green dashboard:

1. A degenerate fit raises, it never scores. Pass three markers (or homography=None) and assert validate_fit returns FitVerdict.DEGENERATE; assert recover_homography raises DegenerateFit when the cache is empty and that no projection was called.
2. The RMS guard bites. Feed correspondences whose fit reprojects at 5.2 px against a 4.0 px guard and assert the verdict is DEGRADED with the reason naming the RMS, and that recover_homography quarantines rather than returning the matrix.
3. The condition gate catches clustered markers. Supply four near-collinear markers that still solve, and assert the verdict is DEGRADED on condition_number even though the RMS looks acceptable.
4. Fallback engages on occlusion. Prime the cache under registry_revision r_2026_07, pass a DEGENERATE frame under the same revision, and assert recover_homography returns the cached matrix with is_degraded=True.
5. A stale cache refuses to serve. Prime the cache under r_2026_06, request a fallback under r_2026_07, and assert cache.get returns None so the frame quarantines instead of scoring on old geometry.
6. Quarantined frames replay correctly. Recalibrate, run replay_fixture on two held captures, and assert the now-passing capture returns FitVerdict.OK while a still-occluded one re-quarantines — and that outcomes come back in capture_timestamp order.

A healthy run shows the degraded-fallback share tracking transient occlusion events and the quarantine share tracking genuine camera-movement incidents, with no fixture scoring continuously on a cached transform for more than a shift.

Troubleshooting Jump to heading#

Related Jump to heading#

• Error Handling in Computer Vision Pipelines — the parent layer whose typed dead-letter and quarantine contract this recovery reuses
• Position Validation Algorithms for Planograms — the stage that fits the homography and scores against it once this gate passes
• Handling Angled-Camera Parallax in Position Scoring — the parallax cause behind many degraded fits, addressed at scoring time