Mike Cassidy

Apr 25, 2017

DIY site search? Hope you have a strong stomach

So, you’d like to build your own search engine. We’d like to save you the trouble. And it is trouble, as you no doubt know.

Of course it’s possible to go DIY on a search engine project. There are powerful starter kits out there — Solr, for instance. You can build a fine site search engine with Solr, provided you have the right people, sufficient time and enough money.

Oh, and a tolerance for risk and opportunity cost. Building your own site search takes time, meaning you are likely losing out on revenue as you design, build and tune your site search engine. As you work out the bugs and as the system you’ve built plugs along, it’s likely the search experience you’re offering will be subpar, meaning dissatisfied customers and the need to win them back once your search engine is running at an acceptable level.

In the first part of this series on building better site search we’ll focus on the role of algorithms in site search. We’ll cover data and infrastructure in subsequent pieces, though the three are woven tightly together.

What do you get with Solr out of the box?

The way to think about building your own site search engine is by thinking about what you get with Solr out of the box. Sure, it’s scalable right off the shelf. It’s also a proven performer.

But think about search and what it takes to power the kind of results that are relevant and personalized down to the one-to-one level. Reaching that optimal level of customer experience requires:

  • sophisticated algorithms
  • vast amounts of data
  • a cloud-based infrastructure that is custom designed for your particular search system.

You know what you won’t find in the bottom of your big, new box of Solr? Sophisticated algorithms, vast amounts of data and the infrastructure you need to build a powerful site search engine.

In fact, developing the algorithms, gathering the data and designing the system to effectively use these to anticipate the intent of digital consumers is what puts the “do” in a do-it-yourself Solr search engine.

Solr on its own isn’t optimized to rank by revenue. It can’t rank by using personalization based on customer intent, behavior and affinities. It’s not designed to provide discovery beyond site search. It doesn’t come loaded with data regarding products, synonyms, buyer intent. It can’t extract content. In fact, it’s fair to say that out-of-the-box Solr will get you about 20 percent of the way to where you need to be to do search right.

If you want a Solr site search engine to do all those necessary things — rank by revenue, personalize, achieve semantic understanding, understand user behavior — you need to tell it how. You need to build the engine’s brain. Or more likely, a team of people needs to build the engine’s brain.

And that’s done with algorithms. Manually building a search engine’s brain takes time. A lot of time — a lot of building and trying and testing.

Red by any other name is, well, red

Take the synonyms for instance. Obviously a robust synonym thesaurus is a key to site search. When a consumer types “crimson, knee-length, spandex, party dress” into a site search box, the system needs to know that for that individual, a Herve Leger, thin-strap, bandage dress with strappy leather harness belt is one of the products that the customer would be highly interested in.

In fact, when we asked consumers to describe that very Herve Leger dress, 500 people came up with a mind-boggling series of combinations that included 129 words for “red,” 275 different descriptions of the belt, 105 descriptions of the length and 216 words to name the occasion at which one would wear the dress.

And you know how a Solr search system knows that? You tell it. Or a team of people you hire works on teaching the machine that “deep rouge” can mean red and that “corset belt” can mean strappy leather harness belt.

Nevermind that there aren’t enough hours in the day for humans to come up with hundreds of variations of a half dozen or so words that consumers might use to describe a dress — wouldn’t you rather they were doing something better with their time?

Thought so.

There is another way. If you’d like to hear more about it, have a watch of the onDemand webinar:  Five reasons why you and DIY search are not meant to be

This is the first topic in a 3 part series, read Part 2 (data) & Part 3 (infrastructure).

Search sign photo by Pleuntje published under Creative Commons license.


2018 Site Search Report

Download the Report
2018 Site Search Report