Shelf Analytics

Sizing Kafka Partitions for Store Capture Throughput

This walkthrough sits under Message Broker Patterns for Capture Events and solves one precise task: choosing a partition count for the capture-event topic that survives a peak reset-window burst while preserving per-fixture ordering and never starving a consumer. The parent cluster owns the topic contract and delivery semantics; this page owns the arithmetic that comes right after — how many partitions, keyed on what? Pick the number too low and consumers cap out at the partition count no matter how many pods you run, so lag climbs through the reset window; pick it too high and you pay for rebalance storms, open file handles, and end-to-end latency you did not need. Worse, a naive key like store_id sends every capture from a flagship superstore to one partition, and that single hot partition lags while the rest sit idle. The fix is a sizing calculation that takes the larger of a throughput requirement and a consumer-parallelism requirement, a composite partition key that spreads a busy store’s fixtures across the log, and a repartitioning plan that does not shear ordering. This page builds that calculation and a skew checker step by step, and each step is independently verifiable.

Partition key choice: store_id alone creates a hot partition, while a composite store_id plus fixture_id key distributes load evenly across the consumer group Two panels share the same topic of four partitions and a consumer group of three. On the left, the partition key is store_id: all capture events from a flagship store hash to partition 0, which fills to overflow and lags, while partitions 1 through 3 stay nearly empty and their consumers idle. On the right, the partition key is a composite of store_id and fixture_id: the same flagship store's many fixtures spread evenly across all four partitions, each partition carries a similar depth, and the three consumers in the group each own a balanced set of partitions with no lag. Per-fixture ordering holds in both because every event for one fixture keeps the same key and therefore the same partition. key = store_id — hot partition key = store_id : fixture_id — even flagship store, many fixtures flagship store, many fixtures store 42 all captures P0 lagging P1 idle P2 idle P3 idle consumer group (3) c-1 overloaded c-2 starved c-3 starved Throughput capped at one partition — lag climbs through the reset window. store 42 per fixture P0 balanced P1 balanced P2 balanced P3 balanced consumer group (3) c-1 · P0 P3 c-2 · P1 c-3 · P2 Every consumer draws a fair share; per-fixture ordering still holds per key.

Prerequisites & Context

Before sizing anything, confirm the following are measured, not guessed. Partition count is cheap to raise and painful to lower, so the inputs matter more than the formula.

  • Runtime: Python 3.11+ with confluent-kafka (or kafka-python) on an admin host that can reach the broker’s bootstrap servers; no special hardware for the sizing math itself.
  • A measured peak, not an average: the events-per-second the capture-event topic sees during a coordinated chain reset, when field teams re-shoot every fixture in a window. This peak often runs 5–10x the daily mean, and it is the number that must fit — sizing to the average guarantees reset-window lag.
  • A per-partition throughput ceiling: the sustained rate one partition plus one consumer can absorb end to end, including the vision-batch handoff. Treat 10 MB/s per partition as a starting ceiling and refine it from your own consumer benchmark; the Async Image Batching for High-Volume Stores stage is usually what sets it, because that is where a capture event turns into real work.
  • A target consumer count: the maximum parallel consumers you expect in the group. A topic’s effective consumer parallelism is capped at its partition count — extra consumers past that sit idle — so this is a hard floor on partitions.
  • A partition key decision: whether ordering must hold per store_id, per fixture_id, or per composite. The capture pipeline needs per-fixture ordering (a fixture’s frames must process in capture order), which as we will see argues for a composite key rather than store_id alone. The topology this topic lives in is described in Designing a Scalable Shelf Analytics Architecture.

A note on terms: a hot partition is one that receives disproportionately more traffic than its peers because the key space is skewed. The goal is a partition count and key that keep every partition within a narrow band of the mean at peak — not merely enough partitions on paper.

Step 1 — Measure Peak and Per-Partition Throughput

Everything downstream depends on two numbers, so pin them empirically. Sample the topic’s produce rate at your finest granularity across a real reset window and take the maximum, not the mean — a partition plan built on the daily average will be under-provisioned exactly when the reset floods it. Then benchmark one consumer against one partition end to end to get the per-partition ceiling; do not assume the broker’s raw byte rate, because the true limit is how fast your consumer can hand a capture event to the vision batch.

from dataclasses import dataclass


@dataclass(frozen=True)
class ThroughputProfile:
    """Measured, not guessed. Peak is the reset-window maximum."""
    peak_events_per_sec: float
    avg_event_size_kb: float
    per_partition_ceiling_mb_s: float = 10.0  # start here; refine from benchmark

    def __post_init__(self) -> None:
        if self.peak_events_per_sec <= 0:
            raise ValueError("peak_events_per_sec must be positive")
        if self.avg_event_size_kb <= 0:
            raise ValueError("avg_event_size_kb must be positive")
        if self.per_partition_ceiling_mb_s <= 0:
            raise ValueError("per_partition_ceiling_mb_s must be positive")

    @property
    def peak_mb_per_sec(self) -> float:
        return self.peak_events_per_sec * self.avg_event_size_kb / 1024.0

Step 2 — Compute Partitions From the Larger of Two Requirements

Partition count is bounded from below by two independent needs, and the answer is the larger of them plus headroom. The throughput requirement is peak megabytes per second divided by the per-partition ceiling. The parallelism requirement is simply the target consumer count, because a consumer group can never run more active consumers than partitions. Take the maximum, then apply a headroom factor so a routine traffic uptick or one lagging broker does not push you back into a resize.

import math


@dataclass(frozen=True)
class PartitionPlan:
    throughput_partitions: int
    parallelism_partitions: int
    headroom_factor: float
    recommended: int

    def explain(self) -> str:
        driver = ("throughput" if self.throughput_partitions
                  >= self.parallelism_partitions else "parallelism")
        return (f"recommend {self.recommended} partitions "
                f"(driver={driver}, headroom={self.headroom_factor:g}x)")


def size_partitions(
    profile: ThroughputProfile,
    target_consumers: int,
    headroom_factor: float = 1.5,
) -> PartitionPlan:
    """partitions = ceil(max(throughput_need, consumer_parallelism) * headroom)."""
    if target_consumers < 1:
        raise ValueError("target_consumers must be >= 1")
    if headroom_factor < 1.0:
        raise ValueError("headroom_factor must be >= 1.0")

    thru = math.ceil(profile.peak_mb_per_sec / profile.per_partition_ceiling_mb_s)
    thru = max(thru, 1)
    par = target_consumers
    recommended = math.ceil(max(thru, par) * headroom_factor)
    return PartitionPlan(thru, par, headroom_factor, recommended)


if __name__ == "__main__":
    profile = ThroughputProfile(
        peak_events_per_sec=4200, avg_event_size_kb=18.0
    )
    plan = size_partitions(profile, target_consumers=12, headroom_factor=1.5)
    print(f"peak = {profile.peak_mb_per_sec:.1f} MB/s")
    print(plan.explain())

Keep the headroom modest — 1.5x is a sane default. Over-provisioning is not free: every partition adds open file handles, replication traffic, controller metadata, and per-partition end-to-end latency, and it lengthens every rebalance. Aim for the smallest count that clears both requirements with room to breathe, and prefer a number that divides evenly by your consumer count so partitions assign in equal shares.

Step 3 — Pick a Composite Key So Flagship Stores Don’t Go Hot

The partition key is where sizing quietly succeeds or fails. Keying on store_id alone feels natural and preserves store ordering, but it collapses every capture from a flagship superstore onto one partition — and that partition lags through the whole reset while the others idle, exactly the left panel of the diagram. Because the capture pipeline only needs ordering per fixture, not per store, a composite store_id:fixture_id key is both correct and far flatter: a busy store’s many fixtures hash across the entire log while every event for a single fixture still lands on the same partition, so capture order per fixture is preserved.

Do not partition by raw wall-clock time or a monotonic counter either — that concentrates the live edge on one partition. Verify the key’s flatness before you ship it with a skew checker over a real sample of keys.

from collections import Counter
from typing import Iterable


def partition_for(key: str, num_partitions: int) -> int:
    """Deterministic assignment mirroring a stable hash partitioner."""
    if num_partitions < 1:
        raise ValueError("num_partitions must be >= 1")
    digest = 0
    for ch in key:
        digest = (digest * 31 + ord(ch)) & 0xFFFFFFFF
    return digest % num_partitions


def capture_key(store_id: str, fixture_id: str) -> str:
    """Composite key: spreads a store's fixtures, keeps per-fixture ordering."""
    if not store_id or not fixture_id:
        raise ValueError("both store_id and fixture_id are required")
    return f"{store_id}:{fixture_id}"


def check_skew(
    keys: Iterable[str], num_partitions: int, tolerance: float = 0.20
) -> dict:
    """Flag hot partitions: any bucket more than tolerance above the mean."""
    counts = Counter(partition_for(k, num_partitions) for k in keys)
    total = sum(counts.values())
    if total == 0:
        raise ValueError("no keys provided")
    mean = total / num_partitions
    loads = {p: counts.get(p, 0) for p in range(num_partitions)}
    hot = {p: n for p, n in loads.items() if n > mean * (1 + tolerance)}
    peak_ratio = (max(loads.values()) / mean) if mean else 0.0
    return {
        "mean_per_partition": round(mean, 1),
        "peak_to_mean_ratio": round(peak_ratio, 3),
        "hot_partitions": hot,
        "balanced": not hot,
    }

Run check_skew over a day of production keys against the count from Step 2. A store_id-only key on a chain with a few dominant stores lights up hot_partitions; the composite key should return balanced with a peak_to_mean_ratio close to 1.0.

Step 4 — Plan Repartitioning Without Breaking Ordering

Adding partitions to a live topic changes the hash mapping, so a key that used to land on partition 3 may now land on partition 7. Any in-flight events for that key already sitting in the old partition can then be consumed out of order relative to the new arrivals — a silent ordering break for that fixture. Never treat --alter --partitions as a routine dial during a reset window.

The safe pattern is to drain, then cut over. Stop producers (or let the group fully catch up so no key has events straddling the change), raise the partition count while consumers are caught up, and only then resume. For a large jump, prefer a new topic sized correctly from the start with a mirror-and-swap: create capture-events.v2 with the target count, dual-write or mirror, let consumers drain v1, then switch the producer alias. This keeps each key’s history contiguous and is the same broker-topology discipline covered by the parent Message Broker Patterns for Capture Events cluster. Size generously in Step 2 precisely so you rarely have to do this.

Verification & Testing

Confirm each property deterministically rather than trusting a dashboard glance:

  1. Parallelism floor holds. Call size_partitions with a tiny peak but target_consumers=12 and assert recommended >= 12 — the consumer count, not throughput, must drive the answer here.
  2. Throughput floor holds. Call it with target_consumers=2 and a peak that needs 9 partitions and assert throughput drives it, with recommended at least ceil(9 * headroom).
  3. The composite key is flat. Run check_skew over a representative day of capture_key(store, fixture) values and assert balanced is True and peak_to_mean_ratio < 1.2.
  4. store_id alone is not flat. Run check_skew over store_id-only keys for a chain with a flagship store and assert hot_partitions is non-empty — this is the regression the composite key fixes.
  5. Ordering is preserved per key. Assert partition_for(capture_key(s, f), n) is stable across calls for the same key, so every frame of one fixture routes to one partition.
  6. No consumer lag at peak. In a staging replay of the reset window, assert consumer-group p95 lag stays p95 lag < 5 s and end-of-window lag returns to 0.

A healthy plan shows every partition within roughly ±20% of the mean depth at peak, lag draining to zero after the reset window closes, and no single consumer pinned at 100% while its peers idle.

Troubleshooting

Symptom Likely root cause Remediation
One partition lags while the rest idle at peak Hot partition from a store_id-only key concentrating a flagship store Switch to the composite store_id:fixture_id key and re-run check_skew; confirm peak_to_mean_ratio drops toward 1.0
Adding consumers past a point does nothing Consumer count now exceeds partition count — extras sit idle Raise partitions to at least the target consumer count via Step 2, then drain-and-cut over per Step 4
Rebalance storms during the reset window Partition count altered live, or consumers flapping so the group reassigns repeatedly Never resize during peak; use a mirror-and-swap topic, and tune session.timeout.ms so a slow consumer is not evicted mid-batch
End-to-end latency crept up after a resize Over-partitioning — too many partitions add metadata, file handles, and per-partition latency Right-size to the smallest count clearing both floors with 1.5x headroom; do not chase a round number far above need
Skew persists even with the composite key A handful of mega-fixtures dominate, or key cardinality is too low for the partition count Add a bounded salt suffix to the busiest fixtures and re-check, or reduce partitions so cardinality comfortably exceeds them
Back to top