Not long ago, “web traffic” meant people — shoppers browsing your site, clicking through categories, and typing queries into a search bar. You could build an analytics dashboard, study conversion funnels, and optimize your experience for the humans moving through it.
But now, the traffic hitting your commerce site is a mix of three fundamentally different audiences: human shoppers, traditional bots and crawlers, and a rapidly growing wave of AI agents — LLM-powered systems that browse, index, and increasingly purchase on behalf of real users. Each of these audiences behaves differently, has different infrastructure footprints, and demands a different response from your platform.
Most commerce technology vendors haven’t caught up. Their caching strategies, rate-limiting logic, and API architectures were designed for a simpler era. As a result, brands are paying for API calls that deliver zero value, experiencing performance degradation during bot spikes, and unknowingly compromising the personalized experiences they invested heavily to build.
At Bloomreach, we’ve been thinking hard about what best-in-class looks like in this new environment. Here’s our perspective.
The 3-Audience Problem
Let’s be specific about who’s actually hitting your search and discovery APIs.
Human shoppers are the audience you’re optimizing for. They browse, they filter, they respond to personalization and merchandising. A well-tuned experience drives conversion. The goal is to serve them the most relevant, most personalized result as fast as possible.
Traditional bots and crawlers like search engine spiders, price comparison scrapers, and competitive intelligence tools have always been part of the web. They’re generally not converting, but the better-behaved ones have legitimate purposes. The problem is volume: when a crawler discovers your faceted navigation and starts systematically requesting every combination of filters, sizes, colors, and sort orders, you can generate millions of unique API calls from a source that will never buy anything.
AI agents are the new variable. As LLM-powered shopping assistants, autonomous purchase agents, and AI-driven product discovery tools proliferate, they represent a traffic category with no real historical precedent. An AI agent might issue highly structured, high-velocity API requests that look nothing like human browsing patterns — and unlike a traditional bot, it might actually be representing a buyer. How you handle this category matters enormously for where commerce is heading.
The mistake most teams make is treating all of this as a single traffic problem with a single solution. But each audience requires a different strategic response.
Caching Is a Powerful Tool — When Applied Correctly
Let’s talk about caching, because it’s one of the most commonly misunderstood levers in API performance management.
The instinct when API volume spikes is to cache everything. That instinct is wrong — at least when applied bluntly — and here’s why: your most valuable experiences are also your most dynamic ones.
If you’re using Loomi’s relevance by segment, real-time segments, A/B testing, or audience-targeted merchandising rules, caching the full API response serves everyone the same content. Essentially, you’ve just neutralized your personalization investment. A shopper who should see a recommendation set tuned to their behavior profile gets the same generic results as an anonymous first-time visitor.
The right framework needs to be layered:
- Cache the anonymous shared layer aggressively. Unfiltered category pages — your top-level men’s, women’s, electronics, home — are high-traffic, low-variation, and extremely cache-friendly. A shopper who lands on your Women’s Dresses page before signing in is seeing the same results as thousands of other anonymous visitors. Cache that, and cache it at the edge. Keep it fresh relative to your catalog update cadence — typically a 15-to-60-minute TTL is the right range, with one hour as the sensible upper bound.
- Keep personalized and experiment-sensitive surfaces dynamic. Anything that depends on who the user is, what segment they’re in, or which test variant they’re seeing should not be flattened into a cached response. This is where your conversion rate actually lives.
- Be surgical about filtered pages. The combinatorial explosion of faceted navigation — filtering by color and size and brand and rating and price range — generates an enormous number of unique URLs. Caching every possible combination is both storage-inefficient and, critically, creates a perfect attack surface for crawler-driven cache fragmentation. The smarter play is to cache only the highest-repeat filter combinations, and control which faceted paths are even crawlable through your SEO and edge rules.
This isn’t so much a compromise between performance and personalization as a recognition that they just require different treatment.
Bot Traffic Is an Edge Problem, Not a Caching Problem
The thing about bot-driven API volume spikes is that no caching strategy will solve them if you don’t address the underlying behavior at the edge.
When an aggressive scraper or a misconfigured AI agent starts generating novel URL combinations at scale, it creates a flood of cache misses. Your origin servers absorb those requests, your API costs rise, and your performance degrades. And, expanding your cache scope to cover more of the bot’s request patterns only increases storage costs and complexity without fixing the root cause.
The right answer is to treat bot traffic as an infrastructure governance problem, not a product discovery problem:
- Rate-limit high-volume, low-value traffic at your CDN or WAF before it ever reaches your API
- Implement request collapsing so that simultaneous cache misses for the same URL trigger only one upstream call
- Separate your bot and human traffic in analytics so you understand your actual human-driven API volume
- Reduce crawlability of deep filter, sort, and pagination combinations through robots.txt and canonical tag discipline
- Serve verified crawlers cached anonymous responses — don’t shut them out entirely, but don’t give them your personalization resources either
For AI agents specifically, the emerging best practice is to distinguish between high-intent agentic traffic — which may represent real commercial activity and deserves real API treatment — and low-intent scraping agents that are essentially building training datasets or competitive intelligence. As the market starts moving more in this direction, we’ll be working with our customers on a case-by-case basis on what works best for their brand.
Cache Keys: The Detail That Makes or Breaks Your Strategy
One tactical element that you really need to pay attention to is the structure of your cache key.
A cache hit only happens when the incoming request matches a stored key. Build your cache keys too broadly, and you serve stale or incorrect results. But build them too narrowly, and your hit rate collapses because every minor variation in query parameters is treated as a unique request.
Good cache key inputs include query or category ID, locale, currency, site or market context, sort order, page number, and any inventory or pricing signal that materially changes the result set.
Bad cache key inputs include session IDs, tracking parameters, referrer strings, UTM values, and any parameter that decorates the URL without changing what gets returned. These are conversion-tracking conveniences, not query-defining variables — don’t let them fragment your cache.
Why This Is a Competitive Wedge
The commerce platforms that win in the next few years will be the ones that figure out how to serve three fundamentally different traffic audiences from a single infrastructure — without compromising the personalized experience for the humans who actually convert.
That requires architectural thinking that goes beyond “cache more” or “rate limit bots.” It requires a clear framework for what should be dynamic, what should be cached, where governance lives, and how the strategy adapts as AI agent traffic grows from a footnote into a primary consideration.
Vendors who haven’t built this framework yet will push customers toward blunt solutions — blanket caching that kills personalization, or permissive API access that creates runaway costs during bot spikes. Neither is ideal at scale.
Bloomreach’s position is that the anonymous shared layer and the personalized layer are separate engineering concerns that deserve separate strategies. The infrastructure to support that layered approach — including edge-level controls, cache key discipline, and feed-cadence-aligned TTLs — is something we help our customers implement throughout deployment.
What’s Next?
The days of designing your search and discovery infrastructure exclusively for human shoppers are over. Bots have always been part of the equation; AI agents are making that reality impossible to ignore.
The brands that take a deliberate, layered approach to this new traffic reality — caching smartly, governing bots at the edge, keeping personalization dynamic, and building toward a framework that can accommodate agentic commerce — will have a structural performance and cost advantage over those who don’t.
If you’re seeing API volume spikes, rising infrastructure costs, or degraded personalization performance, it’s worth asking whether your current stack was built for the traffic environment you’re actually operating in or the one that existed three years ago.
We’d be glad to walk through what a modern approach with Loomi looks like for your specific architecture. Reach out to us to start a conversation.
