Benchmarking Detection Latency on Edge GPUs

This walkthrough sits under Choosing Between YOLOv8, RT-DETR, and EfficientDet for Retail Shelf Detection and solves one precise task: measuring detection latency and throughput correctly on a constrained edge GPU, so an architecture decision rests on numbers rather than vibes. The parent page weighs accuracy, licensing, and operator ergonomics across the three families; this page owns the part that goes wrong most often in practice — the measurement itself. A Jetson-class device will happily report p95 12 ms when the true figure is p95 41 ms, because the wall clock was read before the GPU finished the frame. Get the harness wrong and every downstream comparison inherits the error: you pick RT-DETR over YOLOv8 on a 3 ms gap that is pure timing noise. This page builds a synchronized, warmup-aware timing harness step by step, and each step is independently verifiable against a known-good signal.

Prerequisites & Context Jump to heading#

Before you trust a single number, pin the whole environment. Latency benchmarking is only meaningful when every variable except the one you are sweeping is frozen, so treat the setup below as non-negotiable.

• A fixed test image set: a representative, cached batch of shelf captures — the same store_id mix and resolution distribution you serve in production — decoded to tensors once and kept resident. Never decode JPEG inside the timing loop; disk and codec jitter will swamp the model signal.
• A warmed, clock-fixed device: a Jetson-class edge GPU (Orin, Xavier) with the power mode set to maximum and the fan forced on. Lock the clocks (nvpmodel plus jetson_clocks) so the governor cannot ramp mid-run. An unpinned clock alone can produce a 2x spread between two identical runs.
• Pinned model and runtime versions: exact torch, CUDA, cuDNN, and TensorRT versions recorded alongside every result, plus the frozen model weights and the export precision (fp16, int8). A latency table without a registry_revision and a runtime fingerprint is not reproducible and cannot settle an architecture argument.
• A synchronization primitive: confidence that you can force the host to wait for the GPU. On CUDA that is torch.cuda.synchronize() or CUDA events; without it your wall-clock read returns the moment the kernel was queued, not the moment it finished.
• A thermal readout: access to the on-die temperature and clock counters so you can watch for throttling across a long sweep, the same discipline the edge-deployment tier in Core Architecture for Shelf Analytics applies to in-store compute.

A note on terms: latency is the wall-clock time to process one frame end to end; throughput is frames completed per second under a chosen batch size. They trade against each other, and the whole point of this exercise is to find the batch that maximizes throughput while keeping p95 inside your frame budget.

Step 1 — Warm Up and Fix the Clocks Jump to heading#

The first several inference passes on a cold device pay one-time costs that have nothing to do with steady-state performance: cuDNN autotuning picks kernels, CUDA graphs and TensorRT engines lazily build, and memory pools allocate. If you fold those iterations into your statistics you inflate p50 and destroy p99. Discard them.

Fix the device state before the harness runs — from the shell, not from Python — so the governor is out of the picture:

sudo nvpmodel -m 0          # max power mode (all cores/clocks available)
sudo jetson_clocks          # pin GPU/EMC clocks to max, disable DVFS scaling

Then, inside the harness, run a fixed number of warmup iterations whose results are thrown away. Ten to thirty passes is enough for most detectors; verify empirically in Step 2 by checking that the discarded window is where the slope flattens. The warmup must run the identical code path — same batch size, same resolution, same synchronization — as the measured loop, or it warms the wrong kernels.

Step 2 — Measure p50/p95/p99 With Proper CUDA Synchronization Jump to heading#

This is the step everyone gets wrong. time.perf_counter() reads the host clock, and CUDA kernel launches are asynchronous, so a naive start; model(x); end measures only how long it took to enqueue the work. You must synchronize — either call torch.cuda.synchronize() after inference, or bracket the region with CUDA events and read their elapsed time. The harness below does both correctly, records every per-frame sample, discards the warmup window, and reports true percentiles with error handling around the device calls.

from __future__ import annotations

import statistics
import time
from dataclasses import dataclass
from typing import Callable, Protocol, runtime_checkable

import torch


@runtime_checkable
class Detector(Protocol):
    """Any callable that maps an input batch tensor to detections."""
    def __call__(self, batch: torch.Tensor) -> object: ...


@dataclass(frozen=True)
class LatencyReport:
    batch_size: int
    resolution: tuple[int, int]
    samples_ms: tuple[float, ...]
    p50_ms: float
    p95_ms: float
    p99_ms: float
    throughput_fps: float
    peak_mem_mb: float

    def within_budget(self, budget_ms: float) -> bool:
        """True when the p95 per-frame latency clears the SLA budget."""
        return self.p95_ms <= budget_ms


def _percentile(sorted_ms: list[float], q: float) -> float:
    """Nearest-rank percentile on an already-sorted sample list."""
    if not sorted_ms:
        raise ValueError("no samples to compute a percentile from")
    rank = max(0, min(len(sorted_ms) - 1, round(q / 100.0 * len(sorted_ms)) - 1))
    return sorted_ms[rank]


def benchmark(
    model: Detector,
    batch: torch.Tensor,
    *,
    warmup: int = 20,
    iters: int = 200,
    device: str = "cuda",
) -> LatencyReport:
    """Time a detector on a fixed resident batch with correct GPU sync.

    Per-frame latency = per-batch wall time / batch_size. Warmup iterations
    are run through the identical code path and discarded before any
    percentile is computed.
    """
    if not torch.cuda.is_available():
        raise RuntimeError("CUDA device not available; edge-GPU timing requires it")
    if batch.ndim != 4:
        raise ValueError(f"expected NCHW batch tensor, got shape {tuple(batch.shape)}")

    bs = batch.shape[0]
    res = (int(batch.shape[2]), int(batch.shape[3]))
    torch.cuda.reset_peak_memory_stats()

    with torch.inference_mode():
        for _ in range(warmup):                 # discarded: autotune + engine build
            model(batch)
        torch.cuda.synchronize()                 # ensure warmup fully drained

        per_frame_ms: list[float] = []
        for _ in range(iters):
            start = time.perf_counter()
            model(batch)
            torch.cuda.synchronize()             # wait for the kernel to FINISH
            elapsed_ms = (time.perf_counter() - start) * 1_000.0
            per_frame_ms.append(elapsed_ms / bs)

    ordered = sorted(per_frame_ms)
    p50 = statistics.median(ordered)
    mean_frame_s = (sum(per_frame_ms) / len(per_frame_ms)) / 1_000.0
    peak_mb = torch.cuda.max_memory_allocated() / (1024 ** 2)

    return LatencyReport(
        batch_size=bs,
        resolution=res,
        samples_ms=tuple(per_frame_ms),
        p50_ms=round(p50, 2),
        p95_ms=round(_percentile(ordered, 95), 2),
        p99_ms=round(_percentile(ordered, 99), 2),
        throughput_fps=round(1.0 / mean_frame_s, 1) if mean_frame_s > 0 else 0.0,
        peak_mem_mb=round(peak_mb, 1),
    )

Report the outcome as inline numbers with their conditions attached — p50 12 ms, p95 18 ms, p99 23 ms at batch 1, 640x640, fp16. A bare “18 ms” is not a benchmark; it is a rumor.

Step 3 — Sweep Batch Size and Input Resolution Jump to heading#

A single operating point tells you nothing about the curve. Detection latency scales sub-linearly with batch size until the GPU saturates, then linearly; it scales roughly with pixel count in resolution. Sweep both, holding everything else fixed, and record a LatencyReport per cell so the parent comparison can be read off a grid rather than a single anecdote.

from itertools import product


def sweep(
    build_model: Callable[[], Detector],
    make_batch: Callable[[int, int], torch.Tensor],
    *,
    batch_sizes: tuple[int, ...] = (1, 2, 4, 8, 16),
    resolutions: tuple[int, ...] = (512, 640, 768),
    budget_ms: float = 45.0,
) -> list[LatencyReport]:
    """Grid the detector across batch sizes and square input resolutions."""
    model = build_model()
    reports: list[LatencyReport] = []
    for bs, res in product(batch_sizes, resolutions):
        batch = make_batch(bs, res)
        try:
            report = benchmark(model, batch)
        except torch.cuda.OutOfMemoryError:
            torch.cuda.empty_cache()             # skip infeasible cells, keep sweeping
            continue
        flag = "OK " if report.within_budget(budget_ms) else "OVER"
        print(f"[{flag}] bs={bs:>2} {res}px  "
              f"p50={report.p50_ms} p95={report.p95_ms} p99={report.p99_ms}ms  "
              f"{report.throughput_fps}fps  {report.peak_mem_mb}MB")
        reports.append(report)
    return reports

Read the grid for the knee — the batch where throughput gains flatten while the p95 tail starts climbing. On the device charted above that knee is batch 8: it lifts throughput to 210 fps while holding p95 52 ms, whereas batch 16 buys only 235 fps at the cost of a p99 97 ms tail that blows the frame budget. That single reading is exactly the kind of grounded input the migration from YOLOv8 to RT-DETR in production decision needs, because RT-DETR’s query-based head behaves differently under batching than YOLOv8’s dense anchors.

Step 4 — Record Throughput, Memory, and Watch for Thermal Throttling Jump to heading#

Latency percentiles are half the story. Log throughput_fps and peak_mem_mb from every cell, and — critically for an edge device in a hot back room — sample the die temperature and clock across a long sustained sweep. A device that clears the budget for 30 seconds and then throttles at 85 C will silently miss it in the store. Watch the counters and abort the run if the clock drops off its pinned ceiling.

from pathlib import Path


def read_thermal() -> dict[str, float]:
    """Read Jetson die temperature (C) and GPU clock (MHz) from sysfs."""
    temp_path = Path("/sys/class/thermal/thermal_zone0/temp")
    try:
        temp_c = int(temp_path.read_text().strip()) / 1000.0
    except (OSError, ValueError) as exc:            # non-Jetson host, or no sensor
        raise RuntimeError(f"cannot read thermal zone: {exc}") from exc
    return {"temp_c": round(temp_c, 1)}


def guard_thermal(max_temp_c: float = 80.0) -> None:
    """Raise before a cell runs if the device is already too hot to trust."""
    reading = read_thermal()
    if reading["temp_c"] >= max_temp_c:
        raise RuntimeError(
            f"device at {reading['temp_c']}C >= {max_temp_c}C; "
            "results will reflect throttling, not steady state"
        )

Persist every LatencyReport with its runtime fingerprint and the thermal trace next to it. That artifact — not a screenshot of one fast run — is what makes the architecture choice defensible six months later when someone asks why the edge tier runs RT-DETR at batch 8.

Verification & Testing Jump to heading#

Confirm the harness measures what you think it does before you believe any comparison it produces:

1. Warmup is actually discarded. Log the raw per-frame series with warmup included, then plot it: the first 10–20 samples must be visibly higher and the curve must flatten before the measured window starts. If the whole series is flat, your warmup count is too high; if it is still sloping at the end, it is too low.
2. Synchronization changes the number. Run the loop once with torch.cuda.synchronize() and once without. The unsynchronized run should report an implausibly low, near-constant time (you measured enqueue latency); the synchronized run should be higher and have a real distribution. If the two agree, your sync call is in the wrong place.
3. p95 is stable across runs. Repeat the same cell three times on the fixed image set. The p95 values should agree within a couple of percent (e.g. 52, 53, 51 ms). A wandering p95 means the clocks are unpinned or the device is throttling — fix the environment, not the harness.
4. Latency scales with resolution as expected. Assert that per-frame p50 at 768x768 exceeds 640x640 exceeds 512x512 monotonically, roughly with pixel count. A non-monotonic result signals a caching artifact or a batch that fell out of memory and silently changed shape.
5. Throughput and latency are consistent. For batch 1, throughput_fps should be close to 1000 / p50_ms. A large gap means host-side overhead (data movement, Python) is dominating and your “detection latency” is really pipeline latency — isolate it before comparing models.

A healthy result set shows tight p95 values across repeats, monotonic resolution scaling, and a clear throughput knee that you can point a colleague at without hand-waving.

Troubleshooting Jump to heading#

Related Jump to heading#

• Choosing Between YOLOv8, RT-DETR, and EfficientDet for Retail Shelf Detection — the parent comparison these latency numbers feed into
• Migrating from YOLOv8 to RT-DETR in Production — the production cut-over that this batch/latency curve directly informs
• Core Architecture for Shelf Analytics — where the edge-deployment tier and its in-store compute budget are defined