Preparing for Peak: A Look at Bloomreach Holiday Readiness and ‘Cyber 5’ 2020

The busiest time of the year for US digital commerce is typically just after Thanksgiving, with the so-called 'Cyber ​​5' running through the weekend to 'Cyber ​​Monday'.

Every year the engineering and support teams at Bloomreach gear up to support our customers through this busy season and to handle the large volumes of customer visits that we know are coming.  

Our Preparations for The Holiday Season

To make this important trading period as successful as possible, there are a couple of steps we take to maximize performance:

1. Our preparations for the holiday season start in Q3 before going into “holiday lockdown”

At the end of Q3, our product and engineering teams plan the final release in October. As an agile development team, we have a continuous delivery cycle to production every two weeks throughout the year.

But during the holiday season between the beginning of November to the beginning of January, we shrink this to just a single bug fix release in mid-December. This is opposed to our regular cycle which would have brought in new features with multiple releases.

Consequently, the product and engineering teams must conclude new feature development work before the end of October. This is our last real opportunity to add new features and capabilities before we go into “critical bug fix only” mode.

This year, several resilience and performance improvements were added ahead of our holiday season. For example, building failover for third-party services, where we may use multiple providers if one vendor has failures.

2. We preemptively increase system capacity to meet the higher volumes expected during the season. 

We reach out to customers to ensure we have visibility of key events for sales, discounts and ensure system capacity is prepared. On top of this, we do a retroactive data analysis using past years' volumes to determine how best to scale various parts of our system.

Although we have a multi-tenant architecture, we also have the ability to segment critical parts of our system as isolated clusters. 

3. Finally, we prepare for unexpected production issues during the holidays by running chaos testing and incident fire drills. 

Chaos testing is a set of operations we run against hosts and services in our production environment. These can include traffic control commands at a host level restricting network throughput or killing certain services to test for failover and resilience. 

The engineering team is on hand during these fire drills to ensure that no customer is actually impacted. Our support team will surprise our on-call team to test for response times (fortunately, none of these were needed during Cyber ​​week this year). 

Bloomreach and Cyber ​​Week 2020: Results 

The outcome of all our preparation is that Cyber ​​Week 2020 was a resounding success for Bloomreach and our customers.

Our peak capacity on our backend search APIs reached almost 4,000 QPS (query per second), with global average latencies below just 160ms.  

So far this year, we have seen a dramatic increase in volume in our system since the beginning of the pandemic. Since the beginning of the year, our volume has quintupled to culminate at the 4,000 queries per second peak seen on Cyber ​​Monday.

Compared to 2019, which saw a peak of 1,700 QPS, this is more than double the volume from last year's peak.
 

Bloomreach's ability to manage through the high volume and demands of our customers is the result of not only a highly scalable system architecture, but also preparation, planning and the collaboration of our teams.

Onwards and upwards to another successful and high-volume year in 2021! 

Interested in exploring more data and insights about Cyber ​​Week 2020? Check out our Peak Holiday 2020 Report and our recent Commerce Pulse call where we shared 2020 Holiday eCommerce results and analysis. Learn all about the eCommerce trends, consumer behaviors and factors driving a very different holiday shopping season