Compensating for Specular Glare in Shelf Captures

This walkthrough sits under Error Handling in Computer Vision Pipelines and takes on one narrow, high-frequency failure the parent layer only gates in the abstract: specular glare. A gloss-laminated carton, a freezer-door pane, or an overhead LED strip reflected straight into the lens produces a blown-out highlight that swallows the label beneath it. The detector still fires a box, OCR still returns something, and the pipeline happily emits a verdict — usually a false out-of-stock on a shelf that is fully faced, or a garbled brand read that scores as a misplaced SKU. The blow-out is not noise the model can average away; it is missing information. The correct response is to find the glare, decide whether the pixels are recoverable, correct only what is safe to correct, and — critically — let residual glare lower the confidence of any detection it touches so the frame degrades gracefully instead of firing a crisp, wrong compliance flag. This page builds that path step by step, and each step is independently verifiable.

Prerequisites & Context Jump to heading#

Before wiring this in, confirm the following are already in place. Glare compensation runs inside the preprocessing gate the parent layer describes, so it assumes the frame has already been schema-validated and its capture envelope is intact.

• Runtime: Python 3.11+ with opencv-python and numpy on the CPU preprocessing worker; glare masking is cheap and must never touch a GPU. Budget a few milliseconds per frame at a downsampled working size.
• Colour-normalized input: the frame should already be white-balanced by Shelf Image Preprocessing & Normalization. Glare detection keys off near-clipped, low-saturation pixels; an un-normalized warm cast can shift saturation enough to hide a highlight or invent one.
• A confidence floor to protect: the compliance stage must expose the same score threshold used in Threshold Tuning for Compliance Accuracy. Demotion is only meaningful if a demoted detection actually falls below the floor that suppresses a restock trigger.
• Optional exposure bracket: multi-frame fusion needs 2–3 frames of the same fixture at different exposures, keyed by fixture_id and a tight capture_timestamp window. If your capture app only ships a single frame, fusion is unavailable and that route collapses into inpaint-or-recapture.

A note on terms: specular glare is a mirror-like reflection of a light source, distinct from diffuse over-exposure. It is bright and colourless — that pairing is what separates it from a genuinely white package, and the whole detection strategy hangs on it.

Step 1 — Build a Glare Severity Mask Jump to heading#

Detect glare with two cheap per-pixel tests combined with a logical AND, never brightness alone. A pixel is glare only if it is both near the top of the value channel and near the bottom of the saturation channel: a blown highlight is bright and colourless, whereas a white cereal box is bright but still carries measurable saturation in its print and shadows. Thresholding on value alone is the single most common way teams mistake white packaging for glare and inpaint away a legitimate facing.

Emit a severity mask, not a boolean one, so downstream routing can distinguish a pinpoint hotspot from a sheet of freezer-glass reflection. Report both the coverage fraction and the largest connected blob.

from __future__ import annotations

from dataclasses import dataclass

import cv2
import numpy as np


@dataclass(frozen=True)
class GlareMask:
    mask: np.ndarray          # uint8, 255 where glare
    coverage: float           # fraction of frame flagged, 0..1
    largest_blob_frac: float  # largest connected component, 0..1


def build_glare_mask(
    bgr: np.ndarray,
    value_min: int = 244,
    saturation_max: int = 32,
    work_width: int = 640,
) -> GlareMask:
    """Flag specular glare: near-clipped value AND low saturation.

    Runs on a downsampled copy so the cost stays in single-digit
    milliseconds; the mask is scaled back to the caller's frame size.
    """
    if bgr.ndim != 3 or bgr.shape[2] != 3:
        raise ValueError("expected a 3-channel BGR frame")

    h, w = bgr.shape[:2]
    scale = work_width / float(w)
    small = cv2.resize(bgr, (work_width, max(1, int(h * scale))))
    hsv = cv2.cvtColor(small, cv2.COLOR_BGR2HSV)
    sat, val = hsv[:, :, 1], hsv[:, :, 2]

    raw = ((val >= value_min) & (sat <= saturation_max)).astype(np.uint8) * 255
    # Close pinholes and drop speckle so blobs are countable.
    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
    raw = cv2.morphologyEx(raw, cv2.MORPH_OPEN, kernel)
    raw = cv2.morphologyEx(raw, cv2.MORPH_CLOSE, kernel)

    coverage = float(np.count_nonzero(raw)) / raw.size
    n, _, stats, _ = cv2.connectedComponentsWithStats(raw, connectivity=8)
    largest = 0.0
    if n > 1:  # label 0 is background
        largest = float(stats[1:, cv2.CC_STAT_AREA].max()) / raw.size

    full = cv2.resize(raw, (w, h), interpolation=cv2.INTER_NEAREST)
    return GlareMask(mask=full, coverage=round(coverage, 4),
                     largest_blob_frac=round(largest, 4))

Keep value_min and saturation_max in config, not literals: freezer aisles with cold-white LEDs push both thresholds tighter, while a warm-lit health-and-beauty aisle tolerates a looser saturation ceiling.

Step 2 — Decide the Route: Inpaint, Fuse, or Recapture Jump to heading#

A single mask does not imply a single treatment. Route on the mask’s shape, not just its size, because the three correction strategies fail in different regimes. Small, isolated blobs are safely reconstructed from surrounding texture by inpainting. A large but structured reflection — the classic freezer-glass sheen where the same fixture is legible in a differently-exposed frame — is best resolved by multi-frame exposure fusion. And beyond a coverage ceiling, no correction is honest: too much of the shelf is simply gone, and the only correct output is a recapture request pushed back to the associate, exactly the terminal path the parent error-handling layer already owns.

from enum import Enum


class GlareRoute(str, Enum):
    NONE = "none"
    INPAINT = "inpaint"
    FUSE = "fuse"
    RECAPTURE = "recapture"


def choose_route(
    gm: GlareMask,
    *,
    ignore_below: float = 0.005,
    inpaint_blob_ceiling: float = 0.04,
    recapture_coverage: float = 0.22,
    have_exposure_bracket: bool = False,
) -> GlareRoute:
    """Map a severity mask to a correction strategy."""
    if gm.coverage <= ignore_below:
        return GlareRoute.NONE
    if gm.coverage >= recapture_coverage:
        return GlareRoute.RECAPTURE
    if gm.largest_blob_frac <= inpaint_blob_ceiling:
        return GlareRoute.INPAINT
    # Sizeable, structured reflection: fuse if we can, else recapture.
    return GlareRoute.FUSE if have_exposure_bracket else GlareRoute.RECAPTURE

The recapture_coverage ceiling of 0.22 is deliberately conservative: past roughly a fifth of the frame, inpainting hallucinates facings that were never observed, which is far more dangerous than an honest re-shoot. Treat that dial the way the parent layer treats its degradation floor — tuned against a worst-case promotional-reset frame, not the median store.

Step 3 — Apply the Correction and Mark Affected Detections Jump to heading#

Run the chosen correction, then record which detections still overlap residual glare so the next step can demote them. Inpainting with cv2.INPAINT_TELEA reconstructs small blobs from a dilated boundary; fusion takes the per-pixel best-exposed source across the bracket. Whatever runs, never assume the pixels came back clean — re-run the mask on the corrected frame and carry the residual forward. A detection whose box still intersects meaningful residual glare is not trustworthy no matter how confident the model reported it.

def apply_correction(
    bgr: np.ndarray, gm: GlareMask, route: GlareRoute,
    bracket: list[np.ndarray] | None = None,
) -> np.ndarray:
    """Return a glare-corrected frame for INPAINT or FUSE routes."""
    if route is GlareRoute.INPAINT:
        dilated = cv2.dilate(gm.mask, np.ones((7, 7), np.uint8))
        return cv2.inpaint(bgr, dilated, inpaintRadius=3, flags=cv2.INPAINT_TELEA)

    if route is GlareRoute.FUSE:
        if not bracket:
            raise ValueError("FUSE route requires an exposure bracket")
        stack = [bgr, *bracket]
        # Pick, per pixel, the source with the least specular clipping.
        scores = [cv2.cvtColor(f, cv2.COLOR_BGR2HSV)[:, :, 2] for f in stack]
        best = np.argmin(np.stack(scores, axis=0), axis=0)
        out = np.zeros_like(bgr)
        for i, frame in enumerate(stack):
            out[best == i] = frame[best == i]
        return out

    return bgr  # NONE / RECAPTURE: nothing to correct here

Step 4 — Feed Glare Severity Into the Confidence Floor Jump to heading#

The most important step is not the correction — it is making glare degrade gracefully. Every detection overlapping residual glare gets its confidence multiplied down in proportion to how much of its box is still blown out, so a heavily-glared box slides below the compliance floor and is treated as unresolved rather than firing a crisp out_of_stock_flags entry. This is the difference between a pipeline that fails loud-and-wrong and one that fails quiet-and-honest: a demoted detection routes to QUARANTINED, not to a category manager’s dashboard.

@dataclass(frozen=True)
class Detection:
    sku_id: str
    box: tuple[int, int, int, int]   # x, y, w, h
    confidence: float


def demote_on_residual_glare(
    detections: list[Detection], residual: GlareMask,
    *, floor: float = 0.65, max_penalty: float = 0.6,
) -> list[tuple[Detection, float, bool]]:
    """Scale each detection's confidence by its residual-glare overlap.

    Returns (detection, adjusted_confidence, below_floor) triples so the
    caller can quarantine anything the glare pushed under the floor.
    """
    out: list[tuple[Detection, float, bool]] = []
    m = residual.mask
    for det in detections:
        x, y, w, h = det.box
        crop = m[y:y + h, x:x + w]
        overlap = 0.0 if crop.size == 0 else float(np.count_nonzero(crop)) / crop.size
        adjusted = det.confidence * (1.0 - max_penalty * overlap)
        out.append((det, round(adjusted, 4), adjusted < floor))
    return out

Because demotion is proportional, a facing with a sliver of glare on one corner survives with a small penalty, while one that is half-blown drops out of scope entirely — exactly the graceful curve you want instead of a hard on/off switch.

Verification & Testing Jump to heading#

Confirm each rule deterministically rather than trusting a corrected frame by eye:

1. The mask fires on specular, not on white. Feed a synthetic patch that is bright and colourless (value 250, saturation 10) and assert coverage is non-zero; feed a bright but saturated white-package patch (value 250, saturation 90) and assert coverage stays at 0.0. This is the anti-false-positive guard.
2. Coverage routes correctly. Construct masks at 0.02, 0.10 with a small largest blob, 0.10 with a large blob and no bracket, and 0.30, and assert the routes are INPAINT, INPAINT, RECAPTURE, and RECAPTURE respectively; supply a bracket to the third and assert it flips to FUSE.
3. Corrected OCR recovers. Inpaint a frame whose label was masked and assert the residual mask over that box drops below the original coverage, and that the OCR string for the region now matches the expected_sku where it previously failed.
4. Demotion crosses the floor. Give a detection confidence of 0.80 a box with 50% residual overlap under max_penalty=0.6 and assert the adjusted score is 0.56 and below_floor is True; give the same detection 5% overlap and assert it stays above the floor.
5. Recapture never emits a verdict. Assert a RECAPTURE route produces no corrected frame and that the frame is handed to the parent layer’s terminal recapture path rather than scored.

A healthy run shows glare-driven quarantines dominated by the RECAPTURE and heavy-overlap cases, with inpaint-corrected frames recovering their labels and re-entering scoring above the floor.

Troubleshooting Jump to heading#

Related Jump to heading#

• Error Handling in Computer Vision Pipelines — the parent layer whose quality gate and terminal recapture path this glare logic plugs into
• Shelf Image Preprocessing & Normalization — the upstream white-balance stage the saturation-based glare test depends on
• Threshold Tuning for Compliance Accuracy — how the confidence floor that glare demotion clamps against is reconciled with audits