![](\"https:\/\/www.bloomreach.com\/wp-content\/uploads\/2024\/06\/evaluation-ai-in-article-3.png\")
<\/p>\nWith the influx of new AI capabilities that vendors are adding to their software, ecommerce leaders need a better way to evaluate which tech fits their use cases. There\u2019s no such thing as the best AI for all companies \u2014 there\u2019s only the AI that\u2019s best for you, your data, and your customers.<\/p>\n
Most companies are still using RFPs to evaluate AI tools, but the standard feature checklist (with an explanation of each feature) isn\u2019t effective at measuring whether the AI is delivering actual value. All vendors have access to essentially the same foundational models \u2014 the difference will come from how they have fine-tuned those models and applied them to problems worth solving.<\/p>\n
The normal tactic of asking for case studies and references doesn\u2019t necessarily work either, not only because these features are new and unproven, but also because their results depend heavily on the product and customer data you feed them.<\/p>\n
Instead of trying to make a better RFP, ecommerce leaders should borrow the concept of a golden test set from AI model companies and adapt it for their business. Let\u2019s dive in.<\/p>\n
<\/p>\n
A golden test set is a collection of test scenarios that has been curated by human experts and is used to evaluate AI model performance. The set is designed to cover key scenarios that the AI is designed to solve. So, whenever a company releases a new version of its AI model, it tests the model\u2019s performance against this test set so the company can evaluate progress over time in a standardized and objective way. This makes it easier to make informed decisions about which model is the best to deploy.<\/p>\n
To make this work for ecommerce product discovery, companies should create their own golden set of test questions and scenarios that they can test vendors against. While this is immediately useful for anyone planning to replace their search technology soon, it\u2019ll also be useful for future-proofing your processes. As vendors get more aggressive in reaching out about their latest AI innovations, this will be an easy way to benchmark them against current performance.<\/p>\n
Perhaps most importantly, it\u2019ll help ground you as current and future vendors tell you how they\u2019re incorporating new AI models into their technology. By testing against your own golden set, you cna more easily determine whether the vendor\u2019s technology actually creates business results for you.<\/p>\n
When creating your own version of a golden set for product discovery, you can certainly start from scratch. However, if you\u2019d like some guidance and advice, try using Bloomreach\u2019s recommended set (detailed below) and tweaking it for your needs.<\/p>\n
Most search engines are sophisticated enough to apply different AI strategies to different types of queries. A simple example is head queries vs. long-tail queries. Head queries with one or two words are easy to return results for with semantic or even keyword search, whereas tail queries benefit from broad approaches like vector embeddings.<\/p>\n
Therefore, your golden set should include a broad set of query types that represent the searches that actually happen on your site. You can analyze your search data to figure this out. If you need a starting place, here are some common types of queries:<\/p>\n
Once you\u2019ve analyzed your query types, you\u2019ll need to decide how many examples of each one you want to include in your golden set.<\/p>\n
Your first instinct might be to simply go by percentage. That is, if 80% of your queries are head queries, then that\u2019s how many should appear in your golden set. However, it\u2019s not quite that straightforward.<\/p>\n
Some query types are trickier for AI technology to get right than others, and those are the ones you want to make sure have good representation within your golden set.<\/p>\n
To return to our head queries example \u2014 most search engine technology can handle these well even without sophisticated AI. You should include some in your golden set just to be sure, but you certainly don\u2019t need 80%.<\/p>\n
Here\u2019s how Bloomreach typically creates a golden set when running relevance reports for our customers:<\/p>\n
![](\"https:\/\/www.bloomreach.com\/wp-content\/uploads\/2024\/06\/evaluating-ai-in-article-1.png\")
<\/p>\n
Once your golden set is complete, you can move on to running your evaluation.<\/p>\n
![](\"https:\/\/www.bloomreach.com\/wp-content\/uploads\/2024\/06\/evaluating-ai-in-article-2.png\")
<\/p>\n
Example scorecard:<\/strong><\/b> This demonstrates that, across most categories, Vendor A consistently returns better results than the benchmark.<\/p>\n The above process is somewhat subjective, but then again, so is relevance. What one person considers a good result for their customers, another might find confusing. It\u2019s okay to pick the result that\u2019s biased toward the customer experience you want to achieve for your business.<\/p>\n A note on ranking \u2014 you may be wondering at this point how to evaluate ranking optimizations and tools. All vendors should offer some level of AI to automatically optimize ranking for a particular goal (usually revenue). This is easier to evaluate with an RFP since the vendor should offer specific tools that you can see in a demo or try for yourself in a POC environment. After you test a vendor\u2019s AI against your golden set, you can highlight the weakest areas and ask the vendor to show you how the tools they provide will help you optimize those areas further.<\/p>\n Not all vendors offer the same testing capabilities in pre-sales. Here are some tips for asking a vendor to work with your golden set during the evaluation process.<\/p>\n If you\u2019re offered a sandbox, run the test yourself without configuring any rules. Remember, the goal of the golden test set is to evaluate the core AI. Every solution will offer some options to improve on top of the AI foundation, but by comparing the OOTB without configuration, you can compare the AI performance apples to apples across vendors.<\/p>\n If you\u2019re offered a POC \u2014 something where the vendor stands up a demo site to show your results with their AI \u2014 ask them either to use the golden test or to give you access to the demo so you can do it yourself.<\/p>\n If you\u2019re only offered demos that the vendor controls, try asking if they\u2019ve ever done any internal benchmarks of their AI against any of the scenarios you\u2019ve chosen to focus on. If the answer is no, maybe AI isn\u2019t a core function of that company.<\/p>\n If you\u2019re looking for a new product discovery solution, Bloomreach Discovery<\/a> is your best bet to drive impactful results fast. Powered by Loomi<\/a>, an AI built for ecommerce and fine-tuned with 14+ years of data, Bloomreach Discovery empowers you to improve the metrics that matter most for your business.<\/p>\nWorking With Vendors\u2019 Evaluation Processes<\/h2>\n
Put Your Data to the Test Against Bloomreach Discovery<\/h2>\n