Real-Time Compliance Alerting & Webhooks

Within the Reporting & Compliance Dashboards pillar, real-time alerting is the stage that turns a compliance payload into an action the moment a metric crosses a threshold, instead of waiting for a category manager to open a dashboard and notice. It watches the same typed payload the rest of the reporting layer consumes — compliance_percentage, out_of_stock_flags, misplaced_sku_list, registry_revision, capture_timestamp — and when a fixture drops below its floor it emits a signed webhook to whatever downstream systems need to react: a workforce-management platform that dispatches a reset task, a Slack or Teams channel a merchandising lead watches, or an ERP that opens a replenishment order. The hard part is not detecting the breach; it is delivering the notification exactly-effectively-once to systems that fail, time out, and rate-limit, without flooding those systems during a chain-wide reset. This page defines the alert event and webhook contract, the outbox-plus-worker architecture that delivers it reliably, the retry and dedup configuration that tunes it, the failure modes you will actually hit in production, and the queue and latency budget it has to run inside. The specific problem of what happens when a hundred stores reset the same planogram in the same hour is deep enough to have its own page: Throttling Compliance Alert Storms During Chain Resets.

Concept & Data Contract Jump to heading#

Alerting has two contracts, and keeping them separate is what makes the system reliable. The first is the alert event: an internal, typed record the rule engine produces the instant a compliance metric crosses a configured boundary. The second is the webhook event: the outbound, signed envelope a consumer actually receives over HTTP. The alert event is derived from the same compliance payload the Category Manager Briefing Automation sibling batches into a morning digest — the difference is entirely one of latency and framing. A briefing answers “what happened overnight” in a scheduled roll-up; an alert answers “act now” the moment a fixture breaches, and only for breaches worth interrupting someone over.

An alert is not raised for every low score. It is raised on a transition — a fixture whose compliance_percentage was above its floor on the previous capture and is now below it opens a breach; a fixture that recovers closes one. Encoding the transition rather than the level is what prevents a single degraded fixture from re-alerting on every capture for hours. That transition, plus the fixture identity and the compliance context, becomes the webhook event’s data object.

The webhook event carries three things a naive POST does not. It carries an idempotency_key derived deterministically from the breach identity — store_id, fixture_id, the metric, the capture_timestamp, and the transition direction — so a consumer that receives the same event twice can recognize and discard the duplicate. It carries an event_id and a delivery_attempt counter so retries are distinguishable from genuinely new events. And every delivery is authenticated with an HMAC-SHA-256 signature over the raw body and a timestamp, sent in a header, so a consumer can verify the payload came from the alerting service and was not replayed. A delivered webhook event looks like this:

{
  "event_id": "evt_01HZX8QY7M3K9R2F4B6C",
  "event_type": "compliance.breach.opened",
  "idempotency_key": "STORE-4821::BAY-014-SHELF-03::compliance_percentage::2026-06-28T07:42:11Z::opened",
  "occurred_at": "2026-06-28T07:42:13Z",
  "delivery_attempt": 1,
  "data": {
    "store_id": "STORE-4821",
    "fixture_id": "BAY-014-SHELF-03",
    "planogram_id": "PG-2026-GROCERY-A14",
    "registry_revision": 184,
    "capture_timestamp": "2026-06-28T07:42:11Z",
    "compliance_percentage": 0.71,
    "threshold": 0.85,
    "out_of_stock_flags": 3,
    "misplaced_sku_list": ["00098765432107"]
  }
}

The signature travels in a header, not the body, so the body a consumer signs matches the bytes on the wire exactly: X-Shelf-Signature: t=1751096533,v1=9f2c.... The t= timestamp is inside the signed material, which is what lets a consumer reject a replayed request that arrives outside a tolerance window. The registry_revision and capture_timestamp ride along so a downstream system can tell a breach against the current planogram from stale noise arriving late — the same provenance discipline the payload inherits from Planogram Sync & SKU Mapping Strategies, where the compliance payload originates. Because the source payload is versioned by revision, an alert can always be traced back to the exact scored capture that raised it.

Implementation Architecture Jump to heading#

The tempting implementation — evaluate the rule and POST to the consumer inline, right where the score is computed — is the one that loses events. If the HTTP call fails or the process crashes between committing the score and sending the webhook, the alert is gone with no record that it was owed. The fix is the transactional outbox: the rule engine writes the alert event into an outbox table in the same database transaction that persists the compliance score, and a separate delivery worker drains that table asynchronously. Either both the score and the pending alert commit, or neither does. This is the same broker-decoupling discipline the capture pipeline uses in Message Broker Patterns for Capture Events — the outbox is just the broker pattern applied at the reporting edge, where delivery targets are third-party HTTP endpoints rather than internal topics.

Rule evaluation and the outbox Jump to heading#

Rule evaluation is pure: it takes the current payload and the fixture’s previous state and returns zero or one alert event describing a transition. Writing that event to the outbox is a side effect that shares the score’s transaction.

from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import json


@dataclass(frozen=True)
class AlertEvent:
    event_type: str          # "compliance.breach.opened" | "...closed"
    idempotency_key: str
    data: dict


def evaluate_rule(
    payload: dict,
    prev_compliance: float | None,
    threshold: float,
) -> AlertEvent | None:
    """Emit an alert only on a threshold *transition*, never on every low score.

    Returns None when the fixture stays on the same side of the floor, which is
    what stops one degraded fixture from re-alerting on every capture.
    """
    now = payload["compliance_percentage"]
    was_ok = prev_compliance is None or prev_compliance >= threshold
    is_ok = now >= threshold

    if was_ok and not is_ok:
        direction = "opened"
    elif not was_ok and is_ok:
        direction = "closed"
    else:
        return None  # no transition -> no alert

    key = "::".join([
        payload["store_id"], payload["fixture_id"],
        "compliance_percentage", payload["capture_timestamp"], direction,
    ])
    return AlertEvent(
        event_type=f"compliance.breach.{direction}",
        idempotency_key=key,
        data={**payload, "threshold": threshold},
    )


def enqueue_alert(cursor, event: AlertEvent) -> None:
    """Insert into the outbox in the caller's open transaction.

    The UNIQUE constraint on idempotency_key makes a re-run of the same capture
    a no-op instead of a duplicate row, so the outbox is safe to retry too.
    """
    cursor.execute(
        """INSERT INTO alert_outbox (event_type, idempotency_key, body, status)
           VALUES (%s, %s, %s, 'pending')
           ON CONFLICT (idempotency_key) DO NOTHING""",
        (event.event_type, event.idempotency_key,
         json.dumps(event.data, sort_keys=True)),
    )

The UNIQUE constraint on idempotency_key does double duty: it dedups at the source, so re-scoring the same capture never produces a second outbox row, and it makes the whole enqueue path safe to retry. This is why the key is derived deterministically from breach identity rather than generated randomly — randomness would defeat the dedup.

The delivery worker: retries, backoff, and signing Jump to heading#

The worker claims a batch of pending rows, signs each body, and delivers it. A 2xx marks the row delivered; a failure schedules a retry with exponential backoff and jitter; exhausting the retry budget moves the row to the dead-letter table. Signing happens per attempt because the delivery_attempt counter and occurred_at are part of the body a consumer verifies.

import asyncio
import hashlib
import hmac
import json
import random
import time

import httpx

MAX_ATTEMPTS = 8
BASE_DELAY_S = 2.0
MAX_DELAY_S = 900.0  # 15 min ceiling on a single backoff step


def sign_body(secret: bytes, raw_body: bytes, ts: int) -> str:
    """HMAC-SHA-256 over 't.body' so the signed timestamp blocks replay."""
    mac = hmac.new(secret, f"{ts}.".encode() + raw_body, hashlib.sha256)
    return f"t={ts},v1={mac.hexdigest()}"


def backoff_delay(attempt: int) -> float:
    """Exponential backoff with full jitter, clamped to the ceiling."""
    ceiling = min(MAX_DELAY_S, BASE_DELAY_S * (2 ** (attempt - 1)))
    return random.uniform(0, ceiling)


async def deliver(
    client: httpx.AsyncClient,
    row: dict,
    endpoint: str,
    secret: bytes,
) -> bool:
    """Deliver one event. Return True on success, False to schedule a retry.

    Raises nothing to the caller: transport errors and 5xx are treated as
    retryable, 4xx (other than 429) as permanent so we do not hammer a
    consumer that is rejecting the payload outright.
    """
    body = dict(json.loads(row["body"]))
    envelope = {
        "event_id": row["event_id"],
        "event_type": row["event_type"],
        "idempotency_key": row["idempotency_key"],
        "occurred_at": row["created_at"],
        "delivery_attempt": row["attempt"],
        "data": body,
    }
    raw = json.dumps(envelope, sort_keys=True).encode()
    headers = {
        "Content-Type": "application/json",
        "X-Shelf-Signature": sign_body(secret, raw, int(time.time())),
        "Idempotency-Key": row["idempotency_key"],
    }
    try:
        resp = await client.post(endpoint, content=raw, headers=headers,
                                 timeout=10.0)
    except (httpx.TransportError, httpx.TimeoutException):
        return False  # retryable
    if resp.status_code < 300:
        return True
    if resp.status_code == 429 or resp.status_code >= 500:
        return False  # retryable: backpressure or consumer fault
    # 4xx other than 429: permanent, do not retry a rejected payload
    raise PermanentDeliveryError(resp.status_code)


class PermanentDeliveryError(Exception):
    """Raised on a non-retryable 4xx so the row goes straight to dead-letter."""

Two decisions carry the reliability. First, httpx.AsyncClient is used over a synchronous client because the workload is I/O-bound fan-out — a worker spends almost all of its time waiting on consumer sockets, so async concurrency lets one process hold hundreds of in-flight deliveries without a thread per hook. Second, the status-code policy is explicit: transport errors, 429, and 5xx are retryable because they are transient or backpressure; every other 4xx is permanent, because a consumer returning 400 or 422 will reject the identical payload on every retry, and burning eight attempts on it only delays the events queued behind it. A permanent failure dead-letters immediately.

Production Configuration & Tuning Jump to heading#

Everything that governs delivery behavior lives in versioned configuration so it can be tuned per consumer without a deploy. The retry budget, backoff shape, dedup window, and routing are per-endpoint, because a Slack webhook and an ERP ingestion API have completely different tolerances.

alerting:
  rules:
    compliance_percentage:
      threshold: 0.85          # breach floor; below this opens an alert
      hysteresis: 0.03         # must recover to 0.88 to close, damping flap
  delivery:
    max_attempts: 8            # ~ up to the 15 min backoff ceiling, then DLQ
    base_delay_s: 2.0
    max_delay_s: 900.0
    dedup_window_s: 900        # suppress an identical key seen within 15 min
    request_timeout_s: 10.0
  signing:
    active_secret: "whsec_2026_07"
    previous_secret: "whsec_2026_04"   # accepted during rotation overlap
    rotation_overlap_h: 72
  consumers:
    workforce_mgmt:
      url: "https://wfm.internal/hooks/compliance"
      events: ["compliance.breach.opened"]     # opens only; closes are noise here
      max_concurrency: 16
    slack:
      url: "https://hooks.slack.example/services/T000/B000/xxxx"
      events: ["compliance.breach.opened", "compliance.breach.closed"]
      max_concurrency: 4       # respect Slack per-hook rate limits
    erp:
      url: "https://erp.internal/api/replenishment/webhook"
      events: ["compliance.breach.opened"]
      min_out_of_stock_flags: 2   # only route stock-driven breaches to ERP

The single most consequential knob is hysteresis. Without it, a fixture hovering at exactly the 0.85 floor flaps between opened and closed on consecutive captures, and every flap is a webhook to every routed consumer — a fixture that should raise one alert raises dozens. Requiring recovery to 0.88 before a breach can close absorbs the noise of a metric jittering across the boundary. The dedup_window_s of 900 seconds is a second line of defense: even if the transition logic misfires, an identical idempotency_key seen inside the window is suppressed before it is enqueued.

Per-consumer routing is what keeps each downstream system relevant. The ERP only wants breaches with a real stock cause, so it is gated on min_out_of_stock_flags: 2 and never sees a pure misplacement. The workforce platform wants openings, not closings, because a closed breach is not a task. max_concurrency is set below each consumer’s published rate limit so the worker self-throttles rather than earning a stream of 429s — Slack’s 4 is deliberately conservative. Signing-secret rotation is configuration, not a deploy: an active_secret signs new deliveries while a previous_secret remains valid for a rotation_overlap_h window, so a consumer can roll its verification key without a coordinated cutover. The systemic case of a hundred fixtures breaching at once — where even correct per-fixture behavior sums to a flood — is handled by the throttling controls in Throttling Compliance Alert Storms During Chain Resets.

Failure Modes & Debugging Workflow Jump to heading#

When alerts misbehave, the cause is almost always one of five recurring problems. Work them in this order:

1. Alert storms during chain-wide resets. Symptom: a scheduled reset re-scores hundreds of fixtures within minutes, each transiently breaches while the new planogram is being set, and every one fires — the outbox depth spikes and consumers start returning 429. Reproduce by replaying a reset window’s captures at speed. Fix: apply a reset-aware suppression window keyed on registry_revision changes so a fixture mid-reset is not alerted until it settles, and lean on per-consumer max_concurrency to shape the burst; the dedicated mechanics live in the throttling child page.
2. Duplicate deliveries a consumer did not dedup. Symptom: a downstream system opens two reset tasks for one breach. Root cause is at-least-once delivery meeting a consumer that ignores the Idempotency-Key. Reproduce by killing the worker after a POST succeeds but before the row is marked delivered. Fix: confirm the consumer keys on idempotency_key and not event_id (the latter changes across retries by design), and verify the outbox row is marked delivered in the same commit that records the 2xx so a crash cannot re-send a confirmed event.
3. Consumer 5xx and backpressure. Symptom: one endpoint’s deliveries stall while others flow, and its outbox rows climb through their attempt counts. Reproduce by pointing a consumer at a service returning 503. Fix: verify backoff is actually exponential with jitter (a fixed retry interval synchronizes retries into a thundering herd), check that 429 is treated as retryable and max_concurrency is under the consumer’s limit, and isolate the slow consumer onto its own worker pool so it cannot head-of-line-block healthy endpoints.
4. Clock skew breaking signature verification. Symptom: a consumer rejects valid webhooks with a signature or timestamp-tolerance error, intermittently and only from some workers. Root cause is the signing worker’s clock drifting from the consumer’s, pushing the t= timestamp outside the replay-tolerance window. Reproduce by skewing a worker’s clock by several minutes. Fix: run NTP on every worker host, widen the consumer’s tolerance to a few minutes rather than seconds, and confirm both sides sign and verify over the identical raw bytes — a re-serialized body with reordered keys fails verification even when the clock is fine.
5. At-least-once double-fire on worker restart. Symptom: after a deploy or crash, a small batch of events is delivered a second time. This is the expected cost of the outbox guaranteeing no lost events, not a bug — you cannot have both exactly-once delivery and durability across crashes over plain HTTP. Fix: make it a non-event downstream by keeping consumer-side idempotency correct (see #2); the goal is exactly-once effect, which idempotent consumers deliver even under at-least-once transport.

Keep a labeled repository of dead-lettered events bucketed by these causes. A weekly look at which bucket dominates tells you whether to tighten reset suppression, chase a flaky consumer, or fix a clock — and it is the only durable way to keep the dead-letter queue from silently accumulating events nobody replays.

Scaling & Performance Benchmarks Jump to heading#

Alerting is I/O-bound fan-out, so it scales on concurrency, not CPU. The design target is a delivery p95 < 2 s from outbox insert to a consumer 2xx under normal load, and an outbox depth that returns to near-zero within the capture cadence — a steady-state backlog means the worker pool is undersized or a consumer is throttling. Because each event is independent, throughput scales linearly with worker concurrency until a consumer’s own rate limit becomes the ceiling, which is exactly why max_concurrency is per-consumer: one slow ERP endpoint should never starve the Slack fan-out.

Size the pools against the burst, not the average. Steady-state alert volume is low — most captures produce no transition — but a reset window can turn minutes of quiet into thousands of pending rows. Provision enough async concurrency to drain a reset burst inside the one-minute soft SLA the rest of the reporting layer targets, and partition the outbox drain by consumer so a 503-ing endpoint backs off on its own worker without blocking healthy ones. Watch three signals: outbox depth (the true backlog), per-consumer delivery p95 (the health of each downstream), and dead-letter arrival rate (the rate of giving up). A rising dead-letter rate with flat outbox depth points at a consumer rejecting payloads; a rising outbox depth with a healthy dead-letter rate points at insufficient concurrency.

The dead-letter queue is the pressure-release valve that keeps the system honest. An event that exhausts its 8-attempt budget across a 15-minute backoff ceiling is not dropped — it lands in the dead-letter table with its full history for manual or scripted replay once the consumer recovers. This bounds the worker’s effort per event so a single dead endpoint cannot consume unbounded retries, while still guaranteeing that no breach is ever silently lost. Run the workers on the central reporting tier rather than the store edge — unlike the position-scoring stages that stay near the camera, alerting fans out to internet-facing consumers and belongs where egress and secrets are managed — and keep only the typed event and its delivery history in the outbox, never raw captures or images. At chain scale that keeps alerting a negligible slice of the analytics bill while still emitting a per-breach, per-consumer, signed audit trail.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Throttling Compliance Alert Storms During Chain Resets — the deep dive on suppressing the flood when hundreds of fixtures breach at once during a reset
• Category Manager Briefing Automation — the batched-digest counterpart to real-time alerts, built from the same compliance payload
• Message Broker Patterns for Capture Events — the broker-decoupling discipline the transactional outbox reuses at the reporting edge
• Planogram Sync & SKU Mapping Strategies — the upstream layer that produces the versioned compliance payload alerts are derived from