With great tools comes great responsibility. The era of big data and the tools to immediately analyze and act on it has created the need to master the balance of human experience, intuition and inspiration with machines’ ability to analyze information .

by Mike Cassidy and Romil Shah

As the retail industry is upended by changing consumer habits and rapidly evolving technology, the life of a digital merchandiser has changed in ways that were practically unimaginable even a decade ago.

And that’s good news, right? Of course it is. Think about it: Never before have site merchandisers had the kinds of powerful, data-fueled tools that allow them to apply science to their vast knowledge of the art of picking products, building brands and telegraphing trends.

But with great tools comes great responsibility. The era of big data and the tools to immediately analyze and act on it has created the need to master the balance of human experience, intuition and inspiration with machines’ ability to analyze information at an incredible scale in order to amplify human expertise.

What are we talking about? Think about real life. When things are going well, it’s human nature to want to double-down on that and make things even better — and merchandisers, of course, are human. Under Armour shoes So, say one particular pair of Under Armour shoes abruptly begins to sell at a dizzying rate. The temptation, of course, is to pop the top on your data-driven recommendation engine and tinker with a way to boost more Under Armour products to the top of your page. Hoodies, shorts, backpacks. After all, the brand is on fire. Boosting the brand is a natural impulse, but in some cases it’s the wrong one.

The truth is, that when it comes to a commerce website, like the human body, there are symptoms and there is the disease. There are the outward signs and the internal cause. Those Under Armour shoes? The UA Curry Two? They happen to be the shoes worn by Golden State Warriors point guard Stephen Curry, the superstar team leader of an NBA franchise that recently completed an unprecedented winning regular season and fell one game short of a second consecutive NBA championship.

The increased sales have very little to do with brand and everything to do with who the shoe fits. It may not be the sort of backstory that spills over to Under Armour backpacks or hoodies for that matter. So, a heavy handed boost of all Under Armour products may simply annoy a Nike sneakerhead who is looking for a pair of KDs (which your customer data should have told you since he’s been gobbling up every pair of KDs he can for years).

The hypothetical scenario is not only an illustration of the power of celebrity. It is also a cautionary tale of what can happen when digital merchandisers impulsively give in to the temptation to fiddle with machine-learning systems that were carefully crafted to optimize performance.

And boosting brands is just the beginning. There are plenty of other ways the impulse to manually “fix” an e-commerce site can lead to unintended consequences.

Say you sell a watch from a well-known, top brand and therefore you make the decision to boost it to a top spot on the category page. After all, last year’s similar model was a hot seller. But the surprise is that it is simply not selling. It turns out early customers are not pleased with changes the watchmaker made to the latest iteration of its product. The reviews on the site are brutal. The aggregate customer rating is dismal.

That leads to a situation where the watch is doing well in terms of customer engagement. After all, it’s a new watch from a top brand. There might even have been publicity before its release. It’s possible fans have been waiting for the watch to hit the market. And so, they click. But then they see the reviews and they don’t buy.

You might be tempted to bury the watch and replace it with a product that is converting at a higher rate. More conversions equal more revenue. But rather than manually taking charge, which would mean constantly monitoring and moving the products on that page, would you be better off letting algorithms do more of the work? After all, your carefully crafted algorithm will note the watch’s poor performance, which will cause it to fall lower in the site’s product rankings.

That is not to say that overruling an algorithm is never called for. It is just to say that such changes should be carefully deliberated and considered in the broader context of the whole site and your overall goals.

Site merchandisers today, for instance, rightfully see themselves as responsible for managing the digital experience — the entire experience. They know that the way in which consumers experience their brands can be as important, or even more important, than increasing the sales of a product.

As one who manages the digital experience, you pride yourself on creating a feel, an emotion, an aesthetic. It’s what attracts new customers and keeps other customers coming back. And it’s not something algorithms are good at.

Consider a recommendation engine: All the data in the world might tell you that a significant percentage of men, say 15 percent, who buy a particular designer-label shirt also tend to buy a plush bathrobe. bathrobe brigade But to say to a customer (through your digital site), “Hey, you just bought that nice shirt, you might also want to buy this comfy bathrobe,” is simply a discombobulating experience. There is no art there. That’s the time when a skilled merchandiser might want to overrule an algorithm to offer a high-performing sport coat or tie as a complement to the shirt.

The key is to know when to make a move, so that the best-laid plan doesn’t backfire. It starts with taking a holistic view and appreciating the relationship between human and machine. As a site merchandiser, you and your team decide how to optimize your site. You consider goals and come up with your priorities. Maybe it’s conversion or user engagement. Whatever your priorities, you design your algorithm to optimize for them.

Every change you want to make should correlate with those underlying goals. Left to its own devices, a well-crafted algorithm will learn and consistently work toward achieving your priorities. If you tinker with it in pursuit of some goal beyond your core priorities, you will undoubtedly create unintended consequences and your automated system will suddenly be working at cross-purposes.

In the end, it is counterproductive to isolate one data point or one search query and, based on that, make changes to the work that algorithms are carrying out automatically. A retailer might have more than a million products on its site. Shoppers searching for items have a nearly boundless variety of ways of describing them. How many queries can you reasonably improve by tinkering with the algorithm?

Again, that’s not to say that merchandisers and site optimizers don’t need tools and visibility into the algorithms that drive their site. They do! But those tools should be designed to optimize or strategically tweak a product or category. Those changes should drive specific outcomes in the customer experience with no unintended consequences.

Unless those product and site experts both fully understand the mathematics in an algorithm and have the ability to test their algorithm tweaks, the risk is just too great. And speaking of testing, whether your algorithms are built in-house or by a trusted vendor, testing any changes to that algorithm is critical.

Your quants or the vendors likely spend a great deal of time measuring the impact of every tweak they make (or at least, they damn well better be!). You can’t tweak and test properly unless you have the right data, testing platform and statistically significant sample to pull it off with confidence.

If you don’t have those things, yet still want to fiddle with weights and goals, it’s a “be careful what you wish for” situation. The impulse to jump in and make a change based on insufficient data points is understandable, but dangerous.

It’s the kind of thing that leads to the sort of logic at the core of that old joke: My company is losing 10 cents on every widget we sell. Don’t worry, though, we’ll make it up on volume.

When it comes to tinkering with with your automated ranking weights, don’t be one of those trying to make it up on volume.

Photo of Under Armour shoes by Sandy Dover Creative and men in bathrobes by Paul Downey published under Creative Commons license.

Mike Cassidy is BloomReach’s storyteller. Romil Shah is an engineering manager at BloomReach.