Shelf Image Preprocessing & Normalization

Within the Image Parsing & Computer Vision Workflows section, preprocessing is the deterministic stage that runs before a single tensor reaches a detector: it takes whatever a field associate or a robotic scanner captured — an oblique 12-megapixel phone frame under mixed fluorescent and LED light, a fixed-focal-length robot pass with rolling-shutter skew — and turns it into a geometrically and photometrically consistent frame that every downstream stage can trust. This stage is unglamorous and load-bearing at the same time. If it is even slightly non-deterministic, a confidence regression three stages later is impossible to attribute; if it quietly changes the aspect ratio or crushes small text, the detector and the metric homography inherit a defect they cannot see. This page defines the input/output contract preprocessing enforces, a runnable typed pipeline built on OpenCV and numpy where every step is pure and testable, the per-store-camera configuration that tunes it, the failure modes you will actually hit, and how the stage scales from a single CPU core to a batched GPU tier without changing its output.

Concept & Data Contract Jump to heading#

Preprocessing has one job stated two ways: make every frame look as if it came from the same well-calibrated camera under neutral light, and never lose the information needed to undo that transformation. The second half is what separates a production stage from a notebook script. A detector emits boxes in the coordinate space of the normalized frame, and the metric homography in Position Validation Algorithms for Planograms assumes the frame it projects has already been undistorted and letterboxed. If preprocessing scaled the image by 0.3175 and padded 163 px of neutral gray on top, every one of those numbers has to travel forward so a box can be mapped back to the original capture. A preprocessing stage that transforms pixels but throws away the transform is worse than useless — it silently corrupts geometry downstream.

At the inbound boundary the stage consumes a raw capture: the decoded image bytes plus the ingestion envelope — store_id, fixture_id, device_class, capture_timestamp, and a planogram_id reference. At the outbound boundary it emits a normalized frame (a fixed-size, letterboxed, white-balanced, denoised array in the detector’s expected input space) alongside a capture-metadata record that makes the transform reproducible and invertible. The metadata record looks like this:

{
  "capture_id": "8f3c2a1e-7b4d-4e2a-9c11-5a6b7c8d9e0f",
  "store_id": "US-TX-04821",
  "fixture_id": "GONDOLA-A14-BAY3",
  "device_class": "associate_mobile",
  "capture_timestamp": "2026-06-28T07:14:32Z",
  "preprocess_version": "prep-2026.06.1",
  "source_resolution_px": [4032, 3024],
  "output_resolution_px": [1280, 1280],
  "letterbox": { "scale": 0.3175, "pad_left": 0, "pad_top": 163, "pad_color": [114, 114, 114] },
  "white_balance": { "method": "gray_world", "gain_r": 1.12, "gain_g": 1.0, "gain_b": 0.89 },
  "denoise": { "method": "fast_nl_means", "h": 4.0 },
  "undistort": { "applied": true, "camera_profile": "cam-ceil-wide-01" },
  "sharpness_score": 0.74
}

Two properties of this contract are non-negotiable. First, it is versioned: preprocess_version pins the exact parameter set and code path that produced the frame, so a confidence regression can be attributed to a preprocessing release rather than chased through the detector weights. Second, it is invertible: scale and the pad offsets let any downstream consumer map a box from the 1280 × 1280 normalized frame back to the original 4032 × 3024 capture, and the recorded illuminant gains let a reviewer reconstruct the original colors if a human audit disputes a reading. The sharpness_score is carried through rather than recomputed so the frame remains fully described by its own metadata. Everything else — the detector, the SKU resolver, the homography — is written against this contract and never needs to know which phone or robot produced the pixels.

Implementation Architecture Jump to heading#

The pipeline is a fixed, ordered sequence of pure functions. Purity is a deliberate design choice: each step takes an array and parameters and returns a new array plus the parameters it applied, mutating nothing, so a step can be unit-tested against a golden fixture in isolation and the whole chain is deterministic given a camera_profile and a config version. OpenCV owns the geometric and photometric primitives because its undistort, resize, and fastNlMeansDenoisingColored are battle-tested C++ with predictable numerics; numpy owns the array bookkeeping. The order — undistort, then resize/letterbox, then white balance, then denoise — is not arbitrary. Geometry comes first so distortion is removed before the image is committed to a fixed grid; color correction comes before denoising so the denoiser operates on the balanced signal rather than baking a color cast into its edge-preserving averages.

Ordered pure steps Jump to heading#

from __future__ import annotations

from dataclasses import dataclass

import cv2
import numpy as np


@dataclass(frozen=True)
class CameraProfile:
    """Per store-camera intrinsics and preprocessing overrides."""
    profile_id: str
    camera_matrix: np.ndarray | None      # 3x3 intrinsics, or None to skip undistort
    dist_coeffs: np.ndarray | None        # (5,) lens distortion coefficients
    wb_method: str = "gray_world"
    denoise_h: float = 4.0


@dataclass(frozen=True)
class PreprocessConfig:
    target_size: tuple[int, int] = (1280, 1280)   # (height, width) canvas
    pad_color: tuple[int, int, int] = (114, 114, 114)
    denoise_h: float = 4.0
    version: str = "prep-2026.06.1"


@dataclass(frozen=True)
class NormalizedFrame:
    image: np.ndarray                     # HxWx3 uint8 on the target canvas
    scale: float                          # source -> canvas scale factor
    pad: tuple[int, int]                  # (pad_left, pad_top) in canvas px
    wb_gains: tuple[float, float, float]  # applied (R, G, B) gains
    version: str


def deskew_undistort(image: np.ndarray, profile: CameraProfile | None) -> np.ndarray:
    """Remove lens distortion using calibrated intrinsics; a no-op without a profile."""
    if profile is None or profile.camera_matrix is None or profile.dist_coeffs is None:
        return image
    h, w = image.shape[:2]
    new_mtx, _ = cv2.getOptimalNewCameraMatrix(
        profile.camera_matrix, profile.dist_coeffs, (w, h), alpha=0
    )
    return cv2.undistort(image, profile.camera_matrix, profile.dist_coeffs, None, new_mtx)


def resize_letterbox(
    image: np.ndarray, target: tuple[int, int], pad_color: tuple[int, int, int]
) -> tuple[np.ndarray, float, tuple[int, int], tuple[int, int, int, int]]:
    """Scale preserving aspect ratio, then pad to a fixed canvas. Never distorts."""
    th, tw = target
    h, w = image.shape[:2]
    scale = min(tw / w, th / h)
    nw, nh = int(round(w * scale)), int(round(h * scale))
    resized = cv2.resize(image, (nw, nh), interpolation=cv2.INTER_AREA)
    canvas = np.full((th, tw, 3), pad_color, dtype=np.uint8)
    left, top = (tw - nw) // 2, (th - nh) // 2
    canvas[top:top + nh, left:left + nw] = resized
    return canvas, scale, (left, top), (left, top, left + nw, top + nh)


def white_balance_gray_world(
    image: np.ndarray, content_box: tuple[int, int, int, int] | None, clip_pct: float = 97.5
) -> tuple[np.ndarray, tuple[float, float, float]]:
    """Gray-world balance estimated over the content region, ignoring specular highlights."""
    sample = image
    if content_box is not None:
        l, t, r, b = content_box
        sample = image[t:b, l:r]
    flat = sample.reshape(-1, 3).astype(np.float32)          # BGR
    ceiling = np.percentile(flat, clip_pct, axis=0)           # drop blown highlights
    kept = flat[(flat <= ceiling).all(axis=1)]
    means = kept.mean(axis=0) if len(kept) else flat.mean(axis=0)
    gains = float(means.mean()) / np.clip(means, 1e-6, None)  # BGR gains
    out = np.clip(image.astype(np.float32) * gains[None, None, :], 0, 255).astype(np.uint8)
    r, g, bch = float(gains[2]), float(gains[1]), float(gains[0])
    return out, (r, g, bch)


def denoise(image: np.ndarray, h: float) -> np.ndarray:
    """Edge-preserving denoise. Strength h is capped upstream to protect small text."""
    if h <= 0:
        return image
    return cv2.fastNlMeansDenoisingColored(image, None, h, h, 7, 21)


def normalize_frame(
    image: np.ndarray, profile: CameraProfile | None, config: PreprocessConfig
) -> NormalizedFrame:
    """Run the ordered, deterministic preprocessing pipeline for one capture."""
    if image is None or image.ndim != 3 or image.shape[2] != 3:
        raise ValueError("expected a decoded HxWx3 BGR image")
    if image.dtype != np.uint8:
        raise ValueError("preprocessing expects an 8-bit image; normalize bit depth upstream")

    undistorted = deskew_undistort(image, profile)
    boxed, scale, pad, content_box = resize_letterbox(
        undistorted, config.target_size, config.pad_color
    )
    balanced, gains = white_balance_gray_world(boxed, content_box)
    strength = profile.denoise_h if profile is not None else config.denoise_h
    clean = denoise(balanced, strength)
    return NormalizedFrame(clean, scale, pad, gains, config.version)

Two implementation details carry disproportionate weight. resize_letterbox scales by the minimum ratio and pads the remainder, so the aspect ratio is never altered — a plain cv2.resize to a square would stretch a wide gondola shot and hand the detector boxes whose width-to-height ratio no longer matches the training distribution. And white_balance_gray_world estimates its gains over the content_box only, excluding the neutral letterbox pad that would otherwise drag the gray-world mean toward 114; it also clips the top 2.5% of pixels via the 97.5 percentile so a specular highlight off a glossy package does not fool the illuminant estimate. That highlight-exclusion is the seam where the optional glare-mask hook attaches, feeding the specular regions to the recovery logic in Error Handling in Computer Vision Pipelines rather than letting a blown region distort the whole frame.

Production Configuration & Tuning Jump to heading#

None of the numbers above belong in code. Target resolution, pad color, white-balance method, and denoise strength live in versioned configuration, with a per-store-camera profile layered on top so a ceiling-mounted wide-angle unit and a chest-height cooler camera can differ without a deploy. The base config sets the contract the detector expects; the camera profile overrides only what physically differs about that lens and its lighting:

preprocess:
  version: prep-2026.06.1
  target_size: [1280, 1280]          # square canvas the detector was trained on
  pad_color: [114, 114, 114]         # neutral gray letterbox fill (matches training)
  white_balance:
    method: gray_world               # gray_world | white_patch | learned_illuminant
    clip_percentile: 97.5            # ignore specular highlights when estimating gains
  denoise:
    method: fast_nl_means
    h: 4.0                           # luminance filter strength; raise cautiously
    preserve_min_text_px: 12         # skip denoise if it would blur sub-12px glyphs

camera_profiles:
  cam-ceil-wide-01:
    store_id: US-TX-04821
    fixture_class: dry_grocery_gondola
    dist_coeffs: [-0.283, 0.096, 0.0, 0.0, -0.014]
    denoise_h: 3.0                   # clean sensor, light touch
  cam-cooler-door-02:
    store_id: US-TX-04821
    fixture_class: refrigerated_cooler
    white_balance: { method: white_patch }  # condensation skews gray-world
    denoise_h: 6.0                   # noisier low-light glass, stronger filter

The target resolution of 1280 × 1280 is chosen to match the detector’s training input; changing it without retraining silently shifts the effective object scale and is the single most common self-inflicted accuracy regression. The pad color of 114 is not cosmetic — it matches the neutral fill the detector saw during training, so letterbox borders read as “nothing here” rather than as an edge feature. White-balance method is where per-fixture knowledge pays off: gray-world is a safe default for evenly stocked dry grocery, but a refrigerated cooler’s condensation and blue-shifted LED strips break the gray-world assumption, so those cameras switch to white-patch estimation. The full treatment of choosing and validating an illuminant method per lighting environment is the subject of the child page Correcting White Balance Across Store Lighting. The denoise strength h defaults to 4.0 and is the most dangerous knob on the page: every unit of additional smoothing trades noise for lost high-frequency detail, and shelf-edge price tags and small-font SKU labels live exactly in that high-frequency band, which is why preserve_min_text_px exists to abort denoising when it would erase text the OCR resolver in Bounding Box Extraction & SKU Localization still needs to read.

Failure Modes & Debugging Workflow Jump to heading#

When downstream detections degrade and the detector itself has not changed, preprocessing is the prime suspect. These are the five recurring faults, in the order they are worth checking:

1. Over-aggressive denoise erasing small text. Symptom: barcode reads and label OCR confidence collapse across a whole camera while box detection looks fine, because denoising smooths away the fine strokes OCR depends on. Reproduce by re-running a golden frame with denoise_h stepped from 2.0 to 8.0 and watching OCR confidence fall off a cliff. Fix: lower h, enforce preserve_min_text_px so the step self-aborts on frames with sub-12 px glyphs, and never raise denoise strength to compensate for a noisy sensor without checking the OCR impact first.
2. Wrong aspect ratio breaking boxes. Symptom: detections are systematically stretched or squashed, IoU against ground truth drops uniformly, and the metric homography downstream reports elevated reprojection error. Root cause is almost always a resize that forced a non-square capture into the square canvas instead of letterboxing. Fix: confirm resize_letterbox scales by the minimum ratio and pads the remainder, and assert in a test that a 4032 × 3024 input lands as a scaled region plus symmetric pad, never a stretched fill.
3. Color cast from mixed lighting. Symptom: SKU embedding matches and label reads degrade under mixed fluorescent-and-LED aisles or near cooler glass, where a single global gray-world gain cannot correct a frame lit by two illuminants at once. Reproduce on any capture spanning a lit endcap and a shadowed bay. Fix: switch that camera profile to white-patch or a learned illuminant, exclude specular highlights via the clip_percentile gate, and follow the per-environment tuning in Correcting White Balance Across Store Lighting.
4. Double-normalization. Symptom: colors look washed out or contrast is oddly flat, and gains in the metadata record hover suspiciously close to 1.0 on frames that visibly needed correction. Root cause is a frame that was white-balanced or denoised twice — once by an on-device camera pipeline or an ingestion pre-pass, then again here — so the second pass operates on already-corrected pixels. Fix: make the stage idempotent-aware by keying on preprocess_version and refusing to re-normalize a frame that already carries a metadata record, and disable in-camera auto-white-balance on managed devices so this stage is the single source of truth.
5. Stale or missing camera intrinsics. Symptom: undistort is either a silent no-op (a bumped or newly installed camera whose profile was never calibrated) or visibly over-corrected (a profile applied to the wrong lens), and straight shelf edges bow outward or inward. Reproduce by swapping a dist_coeffs set between two camera profiles. Fix: fail loud when a camera_profile is referenced but absent rather than defaulting to identity, and recalibrate whenever a camera is physically serviced — an out-of-date distortion model corrupts the geometry the homography stage later depends on.

Keep a labeled repository of preprocessing regressions bucketed by these five causes; the dominant bucket over a quarter tells you whether to retune denoise, fix letterboxing, recalibrate white balance, deduplicate a double pass, or recalibrate lenses, and it is the only durable way to keep the same defect from recurring after a config change.

Scaling & Performance Benchmarks Jump to heading#

Preprocessing is CPU-bound and cheap per frame, but at chain scale the aggregate matters. On a single modern CPU core the full pipeline runs in roughly 18–30 ms for a 12-megapixel capture, and denoising dominates that budget — fastNlMeansDenoisingColored is the single most expensive step, often 60% of wall-clock time. The first lever is therefore denoise strength and template window, not parallelism: dropping unnecessary smoothing both speeds the stage and protects small text. Where a camera genuinely needs heavy denoising, the CPU path becomes the bottleneck and a GPU path pays off — cv2.cuda resize and a CUDA non-local-means or bilateral variant move the pipeline into the low-single-digit milliseconds per frame, which is worth it only once you are batching enough frames to amortize host-to-device transfer.

Batching is the second lever and it fits naturally into the section’s broker topology. Preprocessing should consume capture-event batches, undistort and letterbox on CPU in parallel workers, and hand contiguous normalized-frame tensors to the GPU detector tier so the two stages pipeline rather than block each other. The third lever is caching per-camera profiles: intrinsics, the optimal new-camera matrix from getOptimalNewCameraMatrix, and undistortion rectify maps are expensive to compute and identical for every frame from a given camera, so precompute a cv2.initUndistortRectifyMap remap table once per camera_profile and reuse it, turning per-frame undistort into a cheap cv2.remap. Partition preprocessing workers by store_id so a camera’s profile cache stays warm on the worker that handles its frames, and design against the same soft one-minute capture-to-dashboard SLA the rest of the section targets — watch preprocessing consumer lag rather than CPU, because a backlog here almost always signals a chain-wide morning store-walk burst that clears linearly as workers are added. Because the stage is stateless per frame and its output is fully described by the capture-metadata record, scaling it out never changes what any downstream consumer sees.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Correcting White Balance Across Store Lighting — the per-environment how-to for choosing and validating an illuminant method
• Bounding Box Extraction & SKU Localization — the downstream detector and OCR resolver that consume the normalized frame
• Error Handling in Computer Vision Pipelines — where the glare-mask hook routes specular regions for recovery
• Position Validation Algorithms for Planograms — the metric homography stage that assumes an already-normalized frame
• Image Parsing & Computer Vision Workflows — the parent section that wires preprocessing into the end-to-end pipeline