Choosing Between YOLOv8, RT-DETR, and EfficientDet for Retail Shelf Detection

Within the Image Parsing & Computer Vision Workflows section, detector selection is the one architectural choice that is expensive to reverse: the model you pick sets your accuracy ceiling on small facings, your latency floor on an edge GPU, and the annotation budget you will spend for years. A retail shelf is close to the worst case for a general object detector — a single gondola bay can hold 200-plus near-identical facings packed edge to edge, many of them the same SKU in a row, most of them small relative to the frame and partially occluded by the facing in front. YOLOv8, RT-DETR, and EfficientDet each make a different structural bet about that scene, and the bet is baked into the architecture rather than tunable after the fact. This page defines the detection task as a data contract, wraps all three families behind one interface so you can measure them on your own fixtures, walks the configuration knobs that actually move compliance accuracy, scores the three on the axes that matter for a store fleet, and enumerates the failure modes each one brings to dense retail imagery. It is the decision layer that sits behind the Vision Model Routing for Shelf Detection control plane: routing decides which trained detector serves a frame, but this page decides which architecture those detectors should be in the first place.

Concept & Data Contract Jump to heading#

Shelf detection is a bounded task with an unusually hostile input distribution, and naming the boundaries first is what keeps the architecture comparison honest. The inbound boundary is a single normalized frame — a routed image envelope carrying store_id, fixture_id, fixture_class, capture_timestamp, and a decodable full-resolution image — and the outbound boundary is a set of typed detections that the downstream Bounding Box Extraction & SKU Localization stage will resolve to canonical SKUs. The detector’s job is narrow: find every product facing in the frame, put a tight box around it, attach a coarse class and a calibrated confidence, and do nothing else. It does not resolve expected_sku, it does not touch planogram geometry, and it does not know what a slot_id is. That separation is what lets the architecture underneath it be swapped without disturbing anything above.

The output contract is deliberately architecture-agnostic. Whether the boxes came from an anchor-based head, an anchor-free head, or a transformer decoder, a detection record looks identical:

{
  "fixture_id": "BAY-014-SHELF-03",
  "capture_timestamp": "2026-06-28T07:42:11Z",
  "detector_id": "rt-detr-r50-shelf-v3",
  "input_resolution": [1280, 1280],
  "detections": [
    { "box_xyxy": [412, 208, 461, 337], "class": "packaged_food", "confidence": 0.91 },
    { "box_xyxy": [463, 206, 512, 339], "class": "packaged_food", "confidence": 0.88 },
    { "box_xyxy": [514, 205, 560, 340], "class": "packaged_food", "confidence": 0.42 }
  ],
  "detection_count": 3,
  "postprocess": { "conf_threshold": 0.25, "nms_iou": 0.5, "tiled": true }
}

The three architectures make three different structural bets about how those boxes get produced, and each bet has a direct retail consequence. YOLOv8 is a single-stage, anchor-free convolutional detector: it predicts box centers and extents directly on a dense feature grid and resolves overlaps with Non-Maximum Suppression. Its bet is that a fast convolutional backbone plus NMS is enough, which is true until facings pack tightly enough that NMS starts deleting real boxes. RT-DETR is a real-time detection transformer: it replaces NMS with a fixed set of learned object queries that attend over the whole feature map and emit a bounded number of boxes directly. Its bet is that global attention resolves crowding better than NMS ever can — which is exactly the retail failure case — at the cost of a heavier decoder and a hard cap on detections per frame. EfficientDet is a compound-scaled anchor-based detector built on a BiFPN feature-pyramid: its bet is parameter efficiency and a clean accuracy-per-FLOP curve, which ages into middling small-object recall on the densest shelves. The contract is what makes this comparison tractable: because every architecture must fill the same detections array, you can put all three behind one interface and measure them head to head on frames from your own stores rather than trusting a COCO leaderboard that has no 200-facing gondola bays in it.

Implementation Architecture Jump to heading#

The only defensible way to choose is to run all three against the same frames through identical postprocessing, so the harness wraps each family behind a single Detector protocol that returns the typed detection list above. This is the same discipline the routing layer relies on — a stable route()-style contract with swappable implementations — applied one level down to the detectors themselves.

from __future__ import annotations

from dataclasses import dataclass
from typing import Protocol, Sequence

import numpy as np


@dataclass(frozen=True)
class Detection:
    box_xyxy: tuple[float, float, float, float]
    cls: str
    confidence: float


class Detector(Protocol):
    """One interface for every architecture. Implementations own their own
    preprocessing, inference, and NMS/query decoding, but must return boxes in
    original-image pixel coordinates so downstream stages stay detector-blind."""

    detector_id: str

    def detect(self, image: np.ndarray, conf: float, nms_iou: float) -> list[Detection]:
        ...


class YoloV8Detector:
    def __init__(self, weights: str, imgsz: int = 1280) -> None:
        from ultralytics import YOLO  # anchor-free, single-stage CNN + NMS
        self._model = YOLO(weights)
        self._imgsz = imgsz
        self.detector_id = f"yolov8-{weights}"

    def detect(self, image: np.ndarray, conf: float, nms_iou: float) -> list[Detection]:
        r = self._model.predict(
            image, imgsz=self._imgsz, conf=conf, iou=nms_iou, verbose=False
        )[0]
        names = r.names
        return [
            Detection(tuple(map(float, b.xyxy[0])), names[int(b.cls)], float(b.conf))
            for b in r.boxes
        ]


class RtDetrDetector:
    def __init__(self, weights: str, imgsz: int = 1280, max_dets: int = 300) -> None:
        from ultralytics import RTDETR  # transformer decoder, NMS-free
        self._model = RTDETR(weights)
        self._imgsz = imgsz
        self._max_dets = max_dets
        self.detector_id = f"rt-detr-{weights}"

    def detect(self, image: np.ndarray, conf: float, nms_iou: float) -> list[Detection]:
        # nms_iou is accepted for interface parity but unused: the query set
        # already resolves overlaps, so there is no suppression pass to tune.
        r = self._model.predict(
            image, imgsz=self._imgsz, conf=conf, max_det=self._max_dets, verbose=False
        )[0]
        names = r.names
        return [
            Detection(tuple(map(float, b.xyxy[0])), names[int(b.cls)], float(b.conf))
            for b in r.boxes
        ]


def run_matched(
    detectors: Sequence[Detector], image: np.ndarray,
    conf: float = 0.25, nms_iou: float = 0.5,
) -> dict[str, list[Detection]]:
    """Fan the same frame across every architecture with identical thresholds,
    so a difference in the detection list is a difference in the model, not the
    harness. Raises loud if any detector returns nothing on a known-dense frame."""
    out: dict[str, list[Detection]] = {}
    for d in detectors:
        dets = d.detect(image, conf=conf, nms_iou=nms_iou)
        if not dets:
            raise RuntimeError(f"{d.detector_id} returned zero boxes on a shelf frame")
        out[d.detector_id] = dets
    return out

The architectural justification falls out of this harness the moment you feed it a dense bay. The anchor-free YOLOv8 head is the cheapest to run and the easiest to fine-tune, but every box it emits must survive the NMS pass, and NMS decides overlap purely by IoU — it cannot tell two genuinely adjacent facings apart from one box duplicated twice. The transformer RT-DETR head sidesteps that entirely: its object queries are trained to each claim one instance, so two touching facings become two queries rather than one box suppressing the other, which is why it tops the small-object-accuracy axis on crowded shelves. The price is in the detect() signature itself — RT-DETR ignores nms_iou because there is no suppression step, but it caps out at max_det (here 300) queries, so a bay with more facings than queries silently truncates. The anchor-based EfficientDet sits between them: its BiFPN gives clean multi-scale features for a given parameter budget, but predefined anchor shapes fit boxy packaged goods well and thin or irregular facings poorly, and like YOLOv8 it still leans on NMS. Because all three satisfy one interface, the choice reduces to measured evidence on your fixtures rather than architectural taste — which is exactly what Benchmarking Detection Latency on Edge GPUs turns into a reproducible harness.

Production Configuration & Tuning Jump to heading#

The settings that decide whether a detector agrees with a human auditor are the same three across all three families — input resolution, confidence threshold, and overlap handling — but each family responds to them differently, so they belong in versioned config keyed by architecture.

shelf_detection:
  common:
    conf_threshold: 0.25       # keep low-confidence facings; SKU stage re-filters
    input_long_edge: 1280      # below ~1024 small-facing recall collapses
    tiling:
      enabled: true            # slice dense bays, detect per tile, merge
      tile: 960
      overlap: 0.20            # 20% overlap so edge facings survive the seam
      merge_nms_iou: 0.55      # cross-tile de-duplication only
  yolov8:
    nms_iou: 0.50              # suppression aggressiveness on adjacent facings
    max_det: 1000              # a full gondola bay needs a high cap
  rt_detr:
    max_det: 300               # hard query cap — tile before you hit it
    nms_iou: null              # NMS-free; no suppression pass exists
  efficientdet:
    nms_iou: 0.50
    anchor_scales: [0.75, 1.0, 1.3]  # retune anchors to facing aspect ratios
    max_det: 1000

Three numbers do most of the work. The conf_threshold of 0.25 is intentionally permissive: a facing seen at a glancing angle can legitimately score 0.30-0.40, and it is cheaper to pass a marginal box downstream — where the SKU-resolution and consensus logic can reject it — than to drop a real facing here and manufacture a phantom vacancy. The input_long_edge of 1280 is the floor for dense grocery; a facing that occupies 28x40 pixels at 1280 shrinks below the receptive field of any of these detectors at 640, and small-object recall falls off a cliff. The overlap knob is where the architectures diverge hardest: for YOLOv8 and EfficientDet, nms_iou at 0.50 is the single most consequential setting on a packed shelf — raise it toward 0.65 and NMS keeps more of the adjacent boxes it would otherwise merge, at the cost of admitting true duplicates. RT-DETR has no such knob to turn, which is a feature, not a gap; its equivalent risk is the max_det query cap, and the mitigation is tiling. Slicing a 4000px gondola frame into overlapping 960px tiles, detecting each tile, and merging with a light cross-tile de-duplication pass keeps every architecture under its detection ceiling and multiplies effective resolution on the smallest facings. The tuned thresholds here feed the same downstream reconciliation as everything else in the pipeline, so they must stay consistent with the confidence bands documented in Threshold Tuning for Compliance Accuracy — a detector confidence of 0.25 is only meaningful if the compliance layer knows that is where the floor was set.

Comparison Matrix Jump to heading#

The SVG at the top of this page is the decision at a glance; this section is the reasoning behind each score. Read every axis as longer bar is better, and treat the numbers as directional starting points to re-measure on your own fixtures, never as a leaderboard to copy.

• Small-object accuracy — RT-DETR (4.5) leads because its query set resolves crowded, near-identical facings without an NMS suppression step; YOLOv8 (4.0) is close behind once tiled; EfficientDet (3.0) trails because fixed anchor shapes and BiFPN scaling lose the smallest facings first. This is the axis that most directly determines compliance_percentage, so it should carry the most weight for dense grocery and the least for spacious apparel fixtures.
• Edge latency — YOLOv8 (5.0) is the fastest single-stage convolutional path and quantizes cleanly to int8; EfficientDet (3.0) is competitive at the small compound-scale coefficients and slower at the large ones; RT-DETR (2.5) pays for its transformer decoder, and that cost is worst on the constrained edge GPUs that live in stores. Quantify this exactly with the harness in Benchmarking Detection Latency on Edge GPUs.
• Edge-GPU footprint — YOLOv8 (4.5) has the smallest quantized memory and warm-model footprint; EfficientDet (3.5) is parameter-efficient by design; RT-DETR (2.5) carries the largest activation memory, which matters when a store server co-hosts several models.
• Training-data economy — YOLOv8 (4.0) fine-tunes to a new banner on the fewest labelled crops and has the deepest tooling ecosystem; EfficientDet (3.5) is close; RT-DETR (2.5) is the hungriest — transformers need more annotated shelves to converge, which is a real line item when every store format needs its own labelled set.
• License openness — RT-DETR and EfficientDet (both 4.5) ship under permissive Apache-style terms; YOLOv8 (2.0) is copyleft under AGPL, which for a commercial analytics platform is a legal decision, not a technical one, and frequently the deciding factor regardless of the accuracy columns.

The honest reading of the matrix is that there is no dominant row-sweep winner, which is why the Vision Model Routing for Shelf Detection layer exists at all: a fleet can run a quantized YOLOv8 as the cheap default on clean, sparse fixtures and reserve an RT-DETR endpoint for the dense, compliance-critical bays where its small-object edge pays for its latency. The deployment tier that decides where those models physically run — the store-level edge server versus a regional GPU pool — is a Core Architecture for Shelf Analytics concern, and the footprint and latency columns above are exactly the inputs that sizing decision consumes.

Failure Modes & Debugging Workflow Jump to heading#

Each architecture fails on retail shelves in a characteristic way. When detections look wrong, identify which of these you are hitting before you reach for retraining.

1. NMS collapsing adjacent facings (YOLOv8, EfficientDet). Symptom: a row of 6 identical facings returns 4 or 5 boxes, and the deficit tracks how tightly the products are packed. Root cause: NMS deletes a real box because its IoU with a neighbor exceeds nms_iou. Reproduce by lowering nms_iou from 0.50 toward 0.40 and watching the count drop further, confirming suppression is the cause. Fix by raising nms_iou toward 0.60-0.65 and, more durably, tiling the bay so adjacent facings land in separate tiles — or move the densest fixture classes to RT-DETR, which has no suppression pass to misfire.
2. Transformer latency blowing the edge budget (RT-DETR). Symptom: mean accuracy is excellent but p95 inference latency on the store’s edge GPU exceeds the frame budget, and a morning store-walk burst backs the queue up. Root cause: the decoder cost is fixed per frame regardless of how many facings exist, and constrained edge GPUs amplify it. Reproduce by benchmarking at production input_long_edge on the target device, not a datacenter card. Fix by quantizing and compiling the model for the device, lowering resolution only where recall allows, or routing only the fixtures that need RT-DETR’s accuracy to it while everything else takes the YOLOv8 path.
3. EfficientDet small-object recall shortfall. Symptom: EfficientDet posts respectable overall mAP but systematically misses the smallest facings — top-shelf items, thin packages, back-row product — which reads downstream as chronic under-facing. Root cause: anchor shapes and the compound-scaled resolution do not cover the smallest boxes well. Reproduce by bucketing recall by box area and confirming the miss rate concentrates in the smallest bucket. Fix by retuning anchor_scales to the facing aspect ratios in your planograms and raising input_long_edge; if the shortfall persists, the architecture is simply the wrong bet for that fixture density.
4. Domain shift from a new banner or camera generation. Symptom: every architecture’s accuracy drifts down over weeks with no config change, worst on newly onboarded stores. Root cause is not the detector choice — it is that the training distribution has moved away from live captures. Reproduce by comparing the feature distribution of recent frames against the training set. This belongs to Debugging Vision Model Drift in Retail Environments and is resolved by retraining, not by re-picking an architecture; the same drift will hit whichever family you deployed.
5. Truncated detections from the query cap (RT-DETR). Symptom: an unusually dense bay returns exactly max_det boxes and no more, and the missing facings cluster at one end of the shelf. Root cause: the frame contains more instances than the fixed query set can emit. Reproduce by counting detections against the max_det of 300 and checking for the hard ceiling. Fix by tiling before the cap is reached rather than raising max_det, which inflates decoder cost on every frame.

Route the genuinely ambiguous cases — a box that one architecture emits and another suppresses — through the consensus and quarantine paths in Error Handling in Computer Vision Pipelines rather than tuning any single detector into a corner. Keep a labelled error repository bucketed by these five causes; the bucket that dominates tells you whether to retile, re-anchor, re-quantize, retrain, or migrate the architecture outright — and if it is the last, Migrating from YOLOv8 to RT-DETR in Production is the shadow-first playbook for doing it without a compliance-data gap.

Scaling & Performance Benchmarks Jump to heading#

At fleet scale the detector is the compute bottleneck of the whole vision pipeline, so the architecture choice is a direct line item on the analytics bill. The numbers to design against are per-frame latency on the actual edge GPU in the store, effective batch size, and cost per store per day — never datacenter throughput, which flatters RT-DETR and hides the transformer tax that a constrained edge card exposes. A quantized YOLOv8 on a modest edge GPU typically clears a 1280px frame well inside a 40 ms budget and batches efficiently, so a single store server sustains a full morning store-walk of several hundred captures without a queue forming. RT-DETR on the same hardware runs materially slower per frame and benefits less from batching because its decoder cost is fixed rather than amortized, so the honest way to afford it is selective routing — reserve it for the dense, compliance-critical bays and let YOLOv8 carry the sparse, low-density remainder of the fleet. EfficientDet’s cost scales with its compound coefficient, giving a tunable dial between the two.

Batch size is bounded by edge-GPU memory, which is where the footprint axis becomes an operational constraint: a store server co-hosting several models has to fit them all in the same card, and RT-DETR’s larger activation memory can force a smaller batch that erodes its throughput further. Partition the detection queue by store_id — or fixture_id for flagship locations — to preserve capture ordering, and watch consumer lag on that partition rather than raw GPU utilization; a backlog almost always signals a chain-wide reset burst, and because each fixture is independent, adding edge workers clears it linearly. The broker batching and backpressure that keep tail latency flat through that burst are the subject of Async Image Batching for High-Volume Stores, and the offline-resilient variant that keeps detecting when connectivity drops is a Core Architecture for Shelf Analytics concern. Cost discipline mirrors the rest of the platform: run inference on the store-level edge tier, sync only the typed detection records upstream, and let the routing mix — cheap default plus expensive specialist — govern the bill. Getting the architecture and the routing mix right, not squeezing a few milliseconds out of any one model, is what keeps the detection tier affordable across thousands of stores while still feeding the Planogram Sync & SKU Mapping Strategies layer the clean, complete facings it needs to compute a trustworthy compliance_percentage.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Benchmarking Detection Latency on Edge GPUs — the reproducible harness that turns the latency and footprint axes here into device-specific numbers
• Migrating from YOLOv8 to RT-DETR in Production — the shadow-first playbook for switching architectures without a compliance-data gap
• Vision Model Routing for Shelf Detection — the control plane that dispatches each frame to the trained detector this page tells you which architecture to build
• Error Handling in Computer Vision Pipelines — the consensus, quarantine, and drift paths for detections the architectures disagree on
• Core Architecture for Shelf Analytics — the deployment tier that decides where these detectors physically run, sized from the latency and footprint axes
• Image Parsing & Computer Vision Workflows — the parent workflow section this detector-selection decision belongs to