Shelf Analytics

Tuning Confidence Thresholds Against Ground-Truth Audits

This walkthrough sits under Threshold Tuning for Compliance Accuracy and solves one precise task: picking the resolution-confidence floor — the value around 0.62 below which an automated “compliant” verdict is not trusted — by reconciling automated outputs against a labelled human audit set, one floor per category. The parent cluster owns the full cost-weighted machinery across four scoring dimensions; this page owns the narrow, repeatable job a category analyst actually runs each quarter: take last month’s automated verdicts, line them up against what an auditor physically saw on the shelf, sweep the floor, and choose the point where the platform’s compliance score agrees with the human. Set that floor by intuition and every downstream number inherits the guess. Set it by reconciliation and the score becomes defensible slot by slot. Each step below is independently verifiable, and none of it requires retraining a model.

Confidence-floor sweep with a per-category operating point chosen against ground truth The plot sweeps the resolution-confidence floor from left to right. Precision of violation detection rises with a higher floor while recall falls, and the F1 curve forms a hump. A dashed vertical line at a floor of 0.62 marks the chosen operating point for the beverages category, sitting near the F1 peak but pulled slightly toward recall because a missed out-of-stock costs more than a false alert. A side panel shows that reconciling each category against its own audit slice yields different floors: beverages 0.62, tobacco 0.71 where a false alert is cheap and a miss is regulated, and seasonal 0.55 where transient displays tolerate wider bands. A footer notes that the floors are stored as a versioned config and re-audited on a rolling window. Sweep the confidence floor, reconcile against the audit set 0.5 0.75 1.0 metric 0.2 0.4 0.6 0.8 confidence floor θ → precision recall F1 chosen floor θ = 0.62 cost-adjusted F1 peak, pulled toward recall Reconcile per category one audit slice, one floor each beverages 0.62 miss is costly → favor recall tobacco 0.71 regulated → tighter, near-zero miss seasonal 0.55 transient → wider, fewer alerts floors differ → store as versioned config, re-audit on a rolling window Positive class = violation (truth is absent or misplaced). A higher floor flags more slots: recall up, precision down. The chosen floor is the point where the automated compliance score agrees with the auditor at the category's cost ratio.

Prerequisites & Context

This procedure is a reconciliation, not a model change, so what you need is data alignment rather than GPUs. Confirm each of the following exists before you sweep anything.

  • Runtime: Python 3.11+ with numpy; scikit-learn is optional but convenient for the curve. No GPU is required — this runs on the same host that already serves scoring.
  • A labelled audit set: a human-verified ground-truth verdict per slot_id — present, absent, or misplaced — captured by a field auditor within the same window as the images. This is the anchor; everything else is measured against it.
  • Matched automated outputs: for every audited slot_id, the platform’s own verdict and the continuous resolution confidence in [0, 1] that produced it, tagged with category, store_id, expected_sku, and capture_timestamp. Join on slot_id and the capture window, never on time alone.
  • A per-category cost ratio: the relative cost of a false alert (a needless audit dispatched against a compliant slot) versus a miss (a real out-of-stock or misplacement left unflagged). You do not need exact dollars — the ratio is what tilts the operating point.
  • A place to version the result: the floors this page selects must land in the same versioned threshold-set record the parent cluster emits, keyed to a registry_revision, so a floor can never retroactively rewrite a score it did not produce.

A note on framing: the positive class here is a violation. A slot whose ground truth is absent or misplaced is a true positive to catch; a present slot flagged as a violation is a false alert. Raising the confidence floor flags more slots as violations — recall climbs, precision falls — which is exactly the false-alerts-versus-misses trade this page tunes.

Step 1 — Join Automated Verdicts to Ground Truth

The sweep is only as honest as the join underneath it. Line up each audited slot with the automated result that scored the same physical shelf in the same window, then collapse both sides into a binary label: was this slot really a violation, and what confidence did the platform assign to the expected SKU being present? A mismatched or time-skewed join is the single most common way a sweep produces a confident, wrong floor.

from dataclasses import dataclass
from typing import Literal

Verdict = Literal["present", "absent", "misplaced"]


@dataclass(frozen=True)
class AutomatedResult:
    slot_id: str
    category: str
    confidence: float          # resolution confidence the expected SKU is present
    predicted: Verdict


@dataclass(frozen=True)
class AuditLabel:
    slot_id: str
    truth: Verdict             # what the human auditor recorded on the shelf


@dataclass(frozen=True)
class MatchedSlot:
    slot_id: str
    category: str
    confidence: float
    is_violation: bool         # truth is absent or misplaced


def join_to_ground_truth(
    automated: list[AutomatedResult],
    audit: list[AuditLabel],
) -> list[MatchedSlot]:
    """Inner-join automated results to audit labels on slot_id.

    Drops unmatched slots on either side rather than guessing, and refuses
    a confidence outside [0, 1] so a broken upstream score fails loudly.
    """
    by_slot = {a.slot_id: a for a in automated}
    matched: list[MatchedSlot] = []
    for label in audit:
        result = by_slot.get(label.slot_id)
        if result is None:
            continue  # audited slot the platform never scored; not reconcilable
        if not 0.0 <= result.confidence <= 1.0:
            raise ValueError(f"confidence out of range for {label.slot_id}")
        matched.append(
            MatchedSlot(
                slot_id=label.slot_id,
                category=result.category,
                confidence=result.confidence,
                is_violation=label.truth != "present",
            )
        )
    if not matched:
        raise ValueError("no slots matched; check slot_id and capture window alignment")
    return matched

The unmatched-slot count is itself a diagnostic. If a large share of audited slots have no automated counterpart, the problem is upstream coverage or a join key, and no floor will fix it. Two subtler alignment traps hide here. First, join on slot_id plus the capture window, not on timestamp proximity — an auditor who walked the aisle an hour after the capture must still reconcile against the image that actually scored, or you measure the delta between two moments rather than the platform’s accuracy. Second, keep the capture_timestamp on each matched record so a later drift investigation can bucket disagreements by day; a floor that was honest in June and dishonest in July is a time signal you will want to see.

Step 2 — Sweep the Floor and Compute Precision, Recall, F1 Per Category

With clean matched pairs, sweep candidate floors and, at each one, treat “confidence below the floor” as an automated violation flag. Compute precision, recall, and F1 against the ground-truth violation label — and do it per category, because a beverage aisle and a tobacco set have different confidence distributions and different economics. A single chain-wide curve averages those apart and hides the point where each category actually agrees with its auditor.

import numpy as np


@dataclass(frozen=True)
class SweepPoint:
    floor: float
    precision: float
    recall: float
    f1: float


def sweep_category(matched: list[MatchedSlot], grid: np.ndarray) -> list[SweepPoint]:
    """Sweep the confidence floor for one category's matched slots.

    A slot is FLAGGED as a violation when confidence < floor. Metrics are
    computed against the ground-truth violation label.
    """
    conf = np.array([m.confidence for m in matched], dtype=float)
    truth = np.array([m.is_violation for m in matched], dtype=bool)
    if truth.sum() == 0 or (~truth).sum() == 0:
        raise ValueError("audit slice has only one class; sweep is undefined")

    points: list[SweepPoint] = []
    for floor in grid:
        flagged = conf < floor
        tp = int(np.sum(flagged & truth))
        fp = int(np.sum(flagged & ~truth))
        fn = int(np.sum(~flagged & truth))
        precision = tp / (tp + fp) if tp + fp else 0.0
        recall = tp / (tp + fn) if tp + fn else 0.0
        f1 = (2 * precision * recall / (precision + recall)
              if precision + recall else 0.0)
        points.append(SweepPoint(round(float(floor), 3),
                                 round(precision, 4), round(recall, 4), round(f1, 4)))
    return points

Run this over a grid such as np.arange(0.40, 0.91, 0.01). A 0.01 step is fine for a few thousand audited slots; a coarser grid saves nothing here and risks stepping over the true operating point, while a finer one only invents precision the audit sample size cannot support. The output is the raw material behind the curve in the diagram: precision climbing, recall falling, F1 cresting somewhere in the middle. Where that crest lands differs by category, which is the whole reason a single global floor underperforms — a category whose violations score confidently low needs a lower floor to separate them cleanly than one whose damaged or occluded packaging drags legitimate present slots down into the same confidence band.

Step 3 — Pick the Floor at the Operating Point That Matches Business Cost

The F1 peak is a starting point, not the answer. Retail errors are asymmetric: on high-velocity beverages a miss costs far more than a false alert, so the operating point should sit slightly past the F1 peak toward recall; on a regulated category such as tobacco a false alert is cheap and the tolerance for misses is near zero, pulling the floor tighter. Encode that as a cost ratio and let it choose among the swept points. This is the same economic logic the parent cluster applies across all four dimensions — here it is narrowed to the single resolution-confidence floor.

def select_floor(
    points: list[SweepPoint],
    cost_false_alert: float,
    cost_miss: float,
    min_recall: float = 0.80,
) -> SweepPoint:
    """Choose the floor minimizing cost-adjusted error subject to a recall floor.

    cost_miss should exceed cost_false_alert wherever a stockout outweighs a
    needless audit; the recall guard stops a cheap-audit category from being
    tuned into missing genuine violations.
    """
    if cost_false_alert <= 0 or cost_miss <= 0:
        raise ValueError("costs must be positive")

    eligible = [p for p in points if p.recall >= min_recall] or points
    # Rank by an F-beta that leans toward recall as the miss/alert ratio grows.
    beta_sq = cost_miss / cost_false_alert
    def fbeta(p: SweepPoint) -> float:
        num = (1 + beta_sq) * p.precision * p.recall
        den = beta_sq * p.precision + p.recall
        return num / den if den else 0.0
    return max(eligible, key=fbeta)


if __name__ == "__main__":
    grid = np.arange(0.40, 0.91, 0.01)
    # toy beverages slice: violations tend to score low-confidence
    rng = np.random.default_rng(7)
    sample = [
        MatchedSlot(f"S{i}", "beverages",
                    float(np.clip(rng.normal(0.75, 0.12), 0, 1)), False)
        for i in range(400)
    ] + [
        MatchedSlot(f"V{i}", "beverages",
                    float(np.clip(rng.normal(0.50, 0.14), 0, 1)), True)
        for i in range(120)
    ]
    pts = sweep_category(sample, grid)
    chosen = select_floor(pts, cost_false_alert=45.0, cost_miss=120.0)
    print(f"beverages floor={chosen.floor} "
          f"P={chosen.precision} R={chosen.recall} F1={chosen.f1}")

A miss-to-alert ratio of 120 to 45 lands beverages near a floor of 0.62 — past the plain F1 peak, biased toward catching the expensive stockout. Feed tobacco a 10 to 50 ratio and the same code tightens toward 0.71. The floors differ because the economics differ, and that difference is the signal, not noise to be averaged away.

Step 4 — Store Floors as Versioned Config and Schedule a Re-Audit

A floor is a derived artifact, not a constant, so it belongs in versioned configuration alongside the audit window and metrics that justified it — never hard-coded in the scorer. Persist the per-category floors keyed to the registry_revision they were reconciled under, and schedule the next audit so the floors are re-derived before they drift. Deciding when to re-tune versus ride out normal variation is a drift question handled in Detecting Seasonal Compliance Drift with Python; a floor that suddenly disagrees with fresh audits after a packaging change is frequently model drift, diagnosed via Error Handling in Computer Vision Pipelines, not a bad floor.

confidence_floors:
  registry_revision: 184
  reconciled_at: "2026-07-14T06:00:00Z"
  audit_window_days: 90
  re_audit_cron: "0 6 1 */3 *"        # quarterly re-reconciliation
  per_category:
    beverages: { floor: 0.62, precision: 0.93, recall: 0.90, audited_slots: 1840 }
    tobacco:   { floor: 0.71, precision: 0.97, recall: 0.99, audited_slots: 640 }
    seasonal:  { floor: 0.55, precision: 0.88, recall: 0.82, audited_slots: 410 }
  default_floor: 0.60                  # categories below the audit-volume floor
  min_audited_slots: 300               # fall back to default below this count

The min_audited_slots guard matters: a category with a thin audit slice cannot support a bespoke floor without overfitting, so it inherits default_floor until enough ground truth accumulates. Wrap the loader so the scorer reads floors from this record at runtime and updates on the re_audit_cron, never on a code release.

Verification & Testing

Confirm each property deterministically rather than trusting a single printed number:

  1. The chosen floor maximizes the target metric. Assert that no other swept point beats select_floor’s pick on its cost-adjusted F-beta — iterate the full grid and confirm the returned point is the argmax among eligible points.
  2. Monotonic trade-off holds. As the floor rises across the grid, assert recall is non-decreasing and precision is non-increasing on the violation class; a violation of monotonicity means the join or the flag direction is inverted.
  3. Per-category floors differ. Reconcile at least two categories with different cost ratios and assert their selected floors are not equal; identical floors across genuinely different economics signal a global-config leak.
  4. The recall guard binds. Give select_floor a min_recall above every swept point’s recall and assert it degrades to the full set rather than raising, so a strict guard never crashes the quarterly run.
  5. Config is versioned and traceable. Assert the persisted record carries a registry_revision and reconciled_at, and that a floor for a category below min_audited_slots resolves to default_floor, not a fitted value.

A healthy result shows the automated compliance score, recomputed at the chosen floors, landing within a few points of the auditor’s pass rate per category — with residual disagreement concentrated in genuinely ambiguous slots, not a systematic bias in one direction.

Troubleshooting

Symptom Likely root cause Remediation
One floor is applied chain-wide and half the categories disagree with audits A single global floor averages distinct confidence distributions and cost ratios together Sweep and select per category as in Steps 2–3; store per-category floors with a default_floor fallback only for thin slices
Floors that agreed last quarter now over-flag after a packaging refresh Model drift shifted the confidence distribution beneath a fixed floor Re-reconcile against a fresh audit slice; if the reliability shifted, it is a drift problem — trace it via Error Handling in Computer Vision Pipelines before touching the floor
Metrics look excellent but field auditors still disagree Imbalanced audit set — almost all present, too few real violations to estimate recall Stratify sampling so each category’s slice carries enough absent/misplaced truth; enforce min_audited_slots and refuse a single-class sweep
The floor tracks F1 but audit-dispatch cost went up Optimizing the wrong metric — plain F1 ignores the miss-versus-alert asymmetry Drive selection with the cost ratio via the F-beta in select_floor, not the raw F1 peak
Floor swings wildly between re-audits Distinguishing normal variation from real drift by eye Gate re-tuning on the drift signal in Detecting Seasonal Compliance Drift with Python rather than re-fitting every window
Back to top