Correcting White Balance Across Store Lighting

This walkthrough sits under Shelf Image Preprocessing & Normalization and solves one precise task: neutralizing the color cast that mixed store lighting stamps onto every shelf capture, so a SKU looks the same to the detector and the embedding matcher whether it was photographed under warm 2700 K fluorescent, cool 5000 K LED, or a daylight-mixed front aisle. The parent stage owns geometric normalization — deskew, crop, resample; this page owns color. The problem is unglamorous but load-bearing: a red-capped bottle drifts orange under incandescent tubes and pink near a window, and a nearest-neighbour embedding lookup that was trained on catalog renders will happily return the wrong variant. Fix the illuminant once, in linear space, and cache the correction per store-camera so you pay the estimation cost rarely rather than per frame. This page builds that pipeline step by step, and each step is independently verifiable.

Prerequisites & Context Jump to heading#

Before applying this page, confirm the following are already in place. White balance is a color operation, and getting the color space wrong is the single most common reason a correction looks right on a histogram but wrong to the detector.

• Runtime: Python 3.11+ with numpy and opencv-python on the preprocessing host; no GPU is required for this stage, and the per-camera profile keeps the per-frame cost near zero.
• Linear vs sRGB frames: know which encoding your decoded image is in. Camera JPEGs and PNGs are gamma-encoded sRGB. A von Kries diagonal scaling is only physically correct in linear light, so you must linearize before applying gains and re-encode afterward. Applying gains to raw sRGB values is the mistake that leaves a stubborn cast in the shadows.
• A neutral reference (optional but preferred): if a store rig includes a gray card or a known-neutral fixture edge in frame, sample it — it beats any statistical guess. Absent that, the gray-world assumption (the average scene reflectance is achromatic) or a white-patch estimate stands in.
• A per-camera profile store: a small keyed store (Redis, a table, or a JSON blob per rig) mapping store_id + camera to the last good gain vector, its color space, registry_revision, and capture_timestamp. Lighting per fixture is stable across a shift, so you estimate rarely and apply from cache.
• A downstream consumer that cares: consistent color feeds the embedding lookup in Bounding Box Extraction & SKU Localization, where a color-shifted crop is the difference between matching a cherry variant and a cola variant.

Step 1 — Estimate the Illuminant Jump to heading#

The illuminant is the color of the light, expressed as the RGB triple a perfectly neutral surface would produce under it. Estimate it, and the correction is just “map that triple back to gray.” Three estimators cover the field, in ascending order of trust: gray-world (cheapest, assumes the average pixel is neutral), white-patch (assumes the brightest pixels are a white surface reflecting the illuminant), and a sampled reference card (ground truth when one is in frame). Compute a gain vector as the neutral target divided by the estimate, per channel.

from __future__ import annotations

from dataclasses import dataclass
from typing import Literal
import numpy as np

EstimatorName = Literal["gray_world", "white_patch"]


@dataclass(frozen=True)
class Illuminant:
    """Estimated scene illuminant and the per-channel gain that neutralizes it."""
    rgb: tuple[float, float, float]      # linear-light estimate, normalized
    gain: tuple[float, float, float]     # neutral / estimate, per channel
    estimator: EstimatorName
    channel_diversity: float             # 0..1, low => color-dominant scene


def estimate_illuminant(
    linear_rgb: np.ndarray,
    estimator: EstimatorName = "gray_world",
    white_patch_pct: float = 99.0,
) -> Illuminant:
    """Estimate the illuminant from a linear-light image (H, W, 3), float in [0, 1]."""
    if linear_rgb.ndim != 3 or linear_rgb.shape[2] != 3:
        raise ValueError("expected an (H, W, 3) linear-RGB array")
    if linear_rgb.dtype.kind != "f":
        raise TypeError("linearize to float before estimating; got integer sRGB")

    flat = linear_rgb.reshape(-1, 3)
    if estimator == "gray_world":
        est = flat.mean(axis=0)
    else:  # white_patch: mean of the brightest pixels per channel
        thresh = np.percentile(flat, white_patch_pct, axis=0)
        est = np.array([flat[flat[:, c] >= thresh[c], c].mean() for c in range(3)])

    est = np.clip(est, 1e-6, None)
    neutral = float(est.mean())
    gain = neutral / est
    diversity = float(est.min() / est.max())  # near 1 => balanced, near 0 => skewed
    return Illuminant(
        rgb=tuple(round(float(v), 4) for v in est / est.max()),
        gain=tuple(round(float(v), 4) for v in gain),
        estimator=estimator,
        channel_diversity=round(diversity, 4),
    )

The channel_diversity figure is carried forward deliberately: a value near 0.30 means one channel dominates the scene, which is exactly when gray-world lies. Step 4 uses it to hold back.

Step 2 — Apply the Diagonal (von Kries) Correction in Linear Space Jump to heading#

The correction is a diagonal matrix multiply: scale R, G, and B independently by their gains. That is the von Kries model of chromatic adaptation, and it is only valid in linear light. So the operation is a sandwich — linearize sRGB to linear RGB, multiply by the diagonal, re-encode to sRGB. Skip the linearization and the shadows stay tinted while the highlights look fixed, because the gamma curve compresses the correction unevenly.

def srgb_to_linear(srgb: np.ndarray) -> np.ndarray:
    """Decode 8-bit or float sRGB (0..1) to linear light."""
    s = srgb.astype(np.float64) / 255.0 if srgb.dtype == np.uint8 else srgb.astype(np.float64)
    return np.where(s <= 0.04045, s / 12.92, ((s + 0.055) / 1.055) ** 2.4)


def linear_to_srgb(linear: np.ndarray) -> np.ndarray:
    """Encode linear light back to 8-bit sRGB."""
    lin = np.clip(linear, 0.0, 1.0)
    s = np.where(lin <= 0.0031308, lin * 12.92, 1.055 * lin ** (1 / 2.4) - 0.055)
    return np.clip(s * 255.0 + 0.5, 0, 255).astype(np.uint8)


def apply_diagonal(linear_rgb: np.ndarray, gain: tuple[float, float, float]) -> np.ndarray:
    """Multiply each channel by its gain in linear space (the von Kries diagonal)."""
    g = np.asarray(gain, dtype=np.float64)
    if g.shape != (3,) or np.any(g <= 0):
        raise ValueError(f"gain must be three positive values, got {gain!r}")
    return np.clip(linear_rgb * g, 0.0, 1.0)


def white_balance(srgb_bgr: np.ndarray, gain: tuple[float, float, float]) -> np.ndarray:
    """Full sandwich for an OpenCV BGR uint8 frame: linearize, correct, re-encode."""
    rgb = srgb_bgr[..., ::-1]                      # BGR -> RGB
    corrected = apply_diagonal(srgb_to_linear(rgb), gain)
    return linear_to_srgb(corrected)[..., ::-1]   # RGB -> BGR

OpenCV hands you BGR; the channel-flip on both ends is not optional. A gain vector applied to a mislabelled channel order produces a confident, wrong correction — a cast that swaps rather than clears.

Step 3 — Build and Cache a Per-Store-Camera Profile Jump to heading#

Lighting under a given fixture does not change frame to frame, so estimating on every capture is waste and, worse, noise — each frame’s gray-world guess wobbles slightly, and a jittering white balance makes embeddings less stable than no correction at all. Estimate once, cache the gain keyed by store_id and camera alongside the color space and registry_revision, and apply from cache until a lighting shift invalidates it.

import time


@dataclass(frozen=True)
class CameraProfile:
    store_id: str
    camera_id: str
    gain: tuple[float, float, float]
    color_space: str                 # e.g. "sRGB"
    registry_revision: int
    capture_timestamp: float
    channel_diversity: float


class WhiteBalanceProfileStore:
    """In-memory cache of per-store-camera gains; swap for Redis in production."""

    def __init__(self, max_age_s: float = 6 * 3600.0) -> None:
        self._by_key: dict[tuple[str, str], CameraProfile] = {}
        self._max_age_s = max_age_s

    def get(self, store_id: str, camera_id: str) -> CameraProfile | None:
        prof = self._by_key.get((store_id, camera_id))
        if prof is None:
            return None
        if time.time() - prof.capture_timestamp > self._max_age_s:
            return None  # stale: a lighting shift is assumed, force re-estimate
        return prof

    def put(self, prof: CameraProfile) -> None:
        self._by_key[(prof.store_id, prof.camera_id)] = prof

Bump registry_revision whenever a store re-lamps or a rig is remounted so a downstream reader can tell a genuine color change from a pipeline artifact. The max_age_s ceiling is the cheap insurance against a silently drifting profile — after it expires, Step 1 runs again.

Step 4 — Guard Against Over-Correction on Color-Dominant Shelves Jump to heading#

Gray-world assumes the average shelf is neutral. A wall of red cola cases, a produce endcap, or a promotional block of one brand color violates that flatly: the estimator reads the dominant product color as a colored light and tries to cancel it, draining the very color a detector needs. Guard in two moves — clamp each gain into a sane band, and blend the correction toward the identity by a confidence weight derived from the channel_diversity measured in Step 1. Low diversity means low confidence means a lighter touch. Persistent, physically impossible casts are a separate failure class handled in Error Handling in Computer Vision Pipelines alongside glare and exposure faults.

def guard_gain(
    ill: Illuminant,
    clamp: tuple[float, float] = (0.7, 1.4),
    diversity_floor: float = 0.45,
) -> tuple[float, float, float]:
    """Clamp gains and blend toward identity when the scene is color-dominant."""
    lo, hi = clamp
    clamped = np.clip(np.asarray(ill.gain), lo, hi)

    # confidence 0 at the floor, 1 once channels are reasonably balanced
    confidence = float(np.clip(
        (ill.channel_diversity - diversity_floor) / (1.0 - diversity_floor), 0.0, 1.0
    ))
    identity = np.ones(3)
    blended = identity + confidence * (clamped - identity)
    return tuple(round(float(v), 4) for v in blended)


if __name__ == "__main__":
    rng = np.random.default_rng(7)
    # a mildly warm-cast frame: red boosted, blue suppressed
    frame = np.clip(rng.normal(0.5, 0.12, (240, 320, 3)), 0, 1)
    frame[..., 0] *= 1.18   # R up
    frame[..., 2] *= 0.82   # B down

    ill = estimate_illuminant(frame, estimator="gray_world")
    safe_gain = guard_gain(ill)
    print(f"raw gain={ill.gain} diversity={ill.channel_diversity} "
          f"guarded={safe_gain}")

The clamp band (0.7, 1.4) reflects that real store lighting rarely demands more than a 40% per-channel correction; a gain outside it is almost always an estimation failure, not a genuine illuminant.

Verification & Testing Jump to heading#

Confirm each rule deterministically rather than eyeballing a swatch:

1. A neutral patch maps to neutral. Synthesize a flat gray frame, tint it warm with a known (1.18, 1.0, 0.82) multiplier, estimate, and assert the corrected patch has all three channel means within 2 levels of each other on the 0–255 scale.
2. Correction happens in linear space. Run the full sandwich, then run a broken variant that multiplies raw sRGB. Assert the linear path reduces the shadow-region channel spread more than the sRGB path — the sRGB path leaves a measurable cast below mid-gray.
3. Color-dominant shelf is not over-corrected. Feed a frame that is 80% one hue so channel_diversity falls near 0.30, and assert guard_gain returns a vector within 0.05 of the identity (1.0, 1.0, 1.0) — the guard suppresses the false correction.
4. The cache is honoured. Put a profile, get it back within max_age_s, and assert the same gain object is returned without re-estimating; advance the clock past the ceiling and assert get returns None.
5. Embedding match improves. On a labelled crop set, assert the mean cosine distance from each corrected crop to its catalog embedding drops versus the uncorrected crops — the number that actually matters downstream.

A healthy run shows corrected neutral patches collapsing to gray, the guard holding steady on saturated blocks, and the embedding distance improving by a few points without the tail of over-corrected produce crops.

Troubleshooting Jump to heading#

Related Jump to heading#

• Shelf Image Preprocessing & Normalization — the parent cluster that owns geometric normalization before this color stage runs
• Bounding Box Extraction & SKU Localization — the embedding-match consumer that benefits directly from color-consistent crops
• Error Handling in Computer Vision Pipelines — where glare, exposure, and physically impossible casts are handled as failure modes rather than corrections