Migrating from YOLOv8 to RT-DETR in Production

This walkthrough sits under Choosing Between YOLOv8, RT-DETR, and EfficientDet for Retail Shelf Detection and solves one precise task: swapping the live shelf detector from an anchor-based YOLOv8 to an anchor-free transformer, RT-DETR, without regressing a single store’s compliance_percentage. The parent page argues which detector to run; this page owns the far riskier question of how to change the one already in production. The trap is that a detector swap looks like a model file change but is actually an output-contract change: YOLOv8 emits dense anchor boxes that must be run through non-maximum suppression, while RT-DETR is NMS-free and emits a fixed set of set-matched queries whose confidences are calibrated differently. If you promote RT-DETR by editing a weights path, you will double-suppress boxes, shift the score distribution, and watch compliance numbers move for reasons that have nothing to do with real shelf state. The safe path is a shadow evaluation against a frozen eval set, a contract adapter that normalizes both detectors to one schema, a confidence recalibration that holds precision constant, and a staged canary with a working rollback. Each step below is independently verifiable.

Prerequisites & Context Jump to heading#

Before you touch a serving config, confirm these are in place. A detector migration is only as safe as the eval harness behind it, and the harness must exist before the swap, not after a regression appears.

• A frozen labeled eval set. A fixed, versioned set of shelf captures with human-verified boxes and SKU labels, pinned by registry_revision so both detectors are judged on identical ground truth. Small and adjacent facings must be over-represented here, because that is exactly where an anchor-free model’s recall behaves differently from an anchor-based one.
• The existing detection output contract. The typed schema your downstream slot-assignment and scoring stages already consume — box coordinates, confidence, class id, and their coordinate convention. RT-DETR must be adapted to this contract, never the reverse.
• A router that can A/B by store. The Vision Model Routing for Shelf Detection layer must be able to send a given store_id to a chosen detector and run a second detector in shadow without affecting the served result. If the router cannot fan out, build that first.
• Runtime: Python 3.11+, both models exported to the same inference runtime, and a GPU that can carry the shadow model’s added load — validate that headroom against Benchmarking Detection Latency on Edge GPUs before you double the inference cost per capture.

Step 1 — Adapt Both Detectors to One Output Contract Jump to heading#

The two detectors disagree about what a “detection” is before they disagree about anything visual. YOLOv8 returns a dense grid of anchor predictions and expects you to run class-wise non-maximum suppression at some IoU threshold to collapse overlaps. RT-DETR is NMS-free: its decoder emits a fixed number of object queries already de-duplicated by set matching, so running NMS on them a second time silently deletes valid adjacent facings. The adapter’s whole job is to make each raw output emerge as the same normalized Detection record — applying NMS to YOLOv8 and not applying it to RT-DETR — so every downstream stage stays oblivious to which model produced the box.

from __future__ import annotations
from dataclasses import dataclass
from typing import Literal, Sequence
import numpy as np

DetectorName = Literal["yolov8", "rt_detr"]


@dataclass(frozen=True)
class Detection:
    """The common contract every downstream stage consumes."""
    sku_class: str
    x1: float
    y1: float
    x2: float
    y2: float
    confidence: float


@dataclass(frozen=True)
class RawOutput:
    """Whatever a detector emits before contract normalization."""
    boxes_xyxy: np.ndarray      # (N, 4)
    scores: np.ndarray          # (N,)
    class_ids: np.ndarray       # (N,)
    needs_nms: bool             # True for YOLOv8, False for RT-DETR


def _nms(boxes: np.ndarray, scores: np.ndarray, iou_thresh: float) -> list[int]:
    """Greedy IoU suppression; returns kept indices in score order."""
    if len(boxes) == 0:
        return []
    order = scores.argsort()[::-1]
    keep: list[int] = []
    areas = (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])
    while order.size:
        i = int(order[0])
        keep.append(i)
        xx1 = np.maximum(boxes[i, 0], boxes[order[1:], 0])
        yy1 = np.maximum(boxes[i, 1], boxes[order[1:], 1])
        xx2 = np.minimum(boxes[i, 2], boxes[order[1:], 2])
        yy2 = np.minimum(boxes[i, 3], boxes[order[1:], 3])
        inter = np.clip(xx2 - xx1, 0, None) * np.clip(yy2 - yy1, 0, None)
        iou = inter / (areas[i] + areas[order[1:]] - inter + 1e-9)
        order = order[1:][iou <= iou_thresh]
    return keep


def adapt(
    raw: RawOutput, class_map: dict[int, str],
    conf_thresh: float, iou_thresh: float = 0.5,
) -> list[Detection]:
    """Normalize a raw detector output into the common Detection contract."""
    if raw.boxes_xyxy.shape[0] != raw.scores.shape[0]:
        raise ValueError("boxes and scores length mismatch")
    idx = np.where(raw.scores >= conf_thresh)[0]
    if raw.needs_nms:
        # YOLOv8 path: collapse anchor overlaps. RT-DETR skips this entirely.
        idx = np.array([idx[k] for k in _nms(
            raw.boxes_xyxy[idx], raw.scores[idx], iou_thresh)], dtype=int)
    out: list[Detection] = []
    for i in idx:
        cid = int(raw.class_ids[i])
        if cid not in class_map:
            raise KeyError(f"class id {cid} missing from class_map")
        b = raw.boxes_xyxy[i]
        out.append(Detection(class_map[cid], float(b[0]), float(b[1]),
                             float(b[2]), float(b[3]), float(raw.scores[i])))
    return out

The needs_nms flag is the entire safety mechanism: it lives on the raw output, is set once at export time, and guarantees that RT-DETR queries are never suppressed twice. A shared class_map keyed to the same SKU vocabulary keeps the two models’ integer class ids from drifting apart.

Step 2 — Shadow-Run Both and Compare Recall on Hard Facings Jump to heading#

With a common contract, run RT-DETR in shadow on live captures while YOLOv8 keeps serving, and score both against the frozen eval set. The headline metric is not overall mAP — it is recall on small and adjacent facings, because a missed facing turns into a false out-of-stock downstream and corrupts compliance_percentage. Match predictions to ground truth by IoU, count a detection as recovered only when the class also matches, and report the two models side by side so a per-slice regression cannot hide inside an aggregate that improved.

@dataclass(frozen=True)
class EvalResult:
    detector: DetectorName
    recall: float
    precision: float
    small_facing_recall: float
    n_gt: int


def _iou(a: Detection, b: Detection) -> float:
    xx1, yy1 = max(a.x1, b.x1), max(a.y1, b.y1)
    xx2, yy2 = min(a.x2, b.x2), min(a.y2, b.y2)
    inter = max(0.0, xx2 - xx1) * max(0.0, yy2 - yy1)
    area_a = (a.x2 - a.x1) * (a.y2 - a.y1)
    area_b = (b.x2 - b.x1) * (b.y2 - b.y1)
    return inter / (area_a + area_b - inter + 1e-9)


def score(
    preds: Sequence[Detection], truth: Sequence[Detection],
    small_flags: Sequence[bool], name: DetectorName, iou_match: float = 0.5,
) -> EvalResult:
    if len(truth) != len(small_flags):
        raise ValueError("truth and small_flags must align")
    matched, small_hits, small_total = 0, 0, sum(small_flags)
    used: set[int] = set()
    for gt_i, gt in enumerate(truth):
        best_j, best = -1, iou_match
        for j, p in enumerate(preds):
            if j in used or p.sku_class != gt.sku_class:
                continue
            v = _iou(p, gt)
            if v >= best:
                best, best_j = v, j
        if best_j >= 0:
            used.add(best_j)
            matched += 1
            if small_flags[gt_i]:
                small_hits += 1
    recall = matched / max(1, len(truth))
    precision = matched / max(1, len(preds))
    return EvalResult(name, round(recall, 4), round(precision, 4),
                      round(small_hits / max(1, small_total), 4), len(truth))

Run score for both detectors over the whole frozen set and require rt_detr.small_facing_recall >= yolov8.small_facing_recall. Parity on the hard slice is the migration’s real acceptance bar; a gain on easy facings never compensates for a loss on the crowded ones.

Step 3 — Recalibrate the Confidence Threshold to Hold Precision Constant Jump to heading#

RT-DETR’s confidences are not on the same scale as YOLOv8’s, so reusing the old conf_thresh (say 0.25) will move the precision/recall operating point and shift compliance scores for purely numerical reasons. Pick the new threshold by holding precision constant: sweep candidate thresholds, and choose the lowest one whose precision matches the live model’s, then confirm recall did not fall. Freeze that value alongside the weights so it ships as one atomic unit.

def calibrate_threshold(
    raw_batches: list[RawOutput], truths: list[list[Detection]],
    smalls: list[list[bool]], class_map: dict[int, str],
    target_precision: float, grid: Sequence[float],
) -> float:
    """Lowest threshold that still meets the live model's precision."""
    best = grid[-1]
    for t in sorted(grid):
        rs, ps = [], []
        for raw, gt, sm in zip(raw_batches, truths, smalls):
            preds = adapt(raw, class_map, conf_thresh=t)
            r = score(preds, gt, sm, raw.needs_nms and "yolov8" or "rt_detr")
            rs.append(r.recall)
            ps.append(r.precision)
        if float(np.mean(ps)) >= target_precision:
            best = t
            break
    return round(best, 3)

A typical outcome is that RT-DETR holds the live precision at a lower threshold — around 0.30 where YOLOv8 needed 0.45 — while recovering the same or more facings. Because the compliance score is downstream of this operating point, feed the chosen value into the reconciliation loop described in Threshold Tuning for Compliance Accuracy so the detector threshold and the compliance threshold are tuned together, not in isolation.

Step 4 — Staged Canary Rollout by Store With Rollback Jump to heading#

Only after shadow parity and recalibration do you serve RT-DETR to real traffic, and even then only through the router. Advance by store_id in stages — 1%, then 5%, then 25%, then the fleet — and hold at each stage long enough to compare the served compliance_percentage against the same store’s trailing baseline. The router keeps YOLOv8 warm the whole time so rollback is a config flip, not a redeploy. If a canary store’s compliance delta exceeds tolerance, route it back and stop the rollout automatically.

def rollout_decision(
    canary_compliance: float, baseline_compliance: float,
    tolerance_pct: float = 1.5,
) -> Literal["promote", "hold", "rollback"]:
    delta = canary_compliance - baseline_compliance
    if delta < -tolerance_pct:
        return "rollback"
    if abs(delta) <= tolerance_pct:
        return "promote"
    return "hold"   # improbably large positive swing — inspect before trusting

Treat a suspiciously large improvement as a hold, not a win: an unexplained jump usually means the two detectors disagree on what counts as a facing, which is a contract bug, not a real compliance gain.

Verification & Testing Jump to heading#

Confirm each guarantee deterministically before widening the canary.

1. RT-DETR is never double-suppressed. Feed the adapter an RawOutput with needs_nms=False and two boxes at 0.9 IoU; assert both survive. Repeat with needs_nms=True and assert one is suppressed.
2. Parity or better recall. Assert rt_detr.small_facing_recall >= yolov8.small_facing_recall on the frozen eval set, and that overall recall did not drop.
3. Precision held at the new threshold. Assert calibrate_threshold returns a value whose mean precision is within 0.01 of the live model’s, and that recall at that value is not lower than before.
4. Compliance delta inside tolerance. On the canary cohort, assert abs(canary_compliance - baseline_compliance) <= 1.5 for each store_id before promoting the next stage.
5. Rollback works. Force a synthetic -3.0 delta and assert rollout_decision returns "rollback" and that the router restores YOLOv8 for that store on the next capture, verified end to end, not just in the unit test.

Troubleshooting Jump to heading#

Related Jump to heading#

• Choosing Between YOLOv8, RT-DETR, and EfficientDet for Retail Shelf Detection — the parent decision this migration acts on
• Vision Model Routing for Shelf Detection — the router that A/Bs and shadows detectors by store and makes rollback a config flip
• Threshold Tuning for Compliance Accuracy — where the recalibrated confidence threshold is reconciled against downstream compliance scoring