Query Triage: The Secret Weapon for Search Relevance

Remember way back in my first post on search relevance how I said there’s a statistical and a human-centered approach to search improvement? Well this post is about one way you can merge your human and metrics-driven approaches to maximize the value of each. This secret weapon is called a Query Triage.

The first thing to understand is that while there are two distinct approaches to search relevance improvement, but neither really lives in a bubble… unless your organization is dysfunctional. The human-centered approach relies on an understanding of technology as well as user needs, and a metrics-driven approach is useless without understanding how those metrics are grounded in and driven by the user experience.

However, it’s all-too-easy for us as search people (whether data scientists, engineers, PMs, UXers, etc) to get stuck in our own habits and our own ways of thinking. It gets difficult for us to see the random forests for the gradient boosted decision trees (that’s a machine learning joke XD.). So to combat this complacency and ensure that the team (as a whole) is aligned on customer problems and how they can be solved, I’m proposing you add a query triage meeting to your repertoire.

On its face, a query triage meeting is quite simple: get a bunch of people into a room and discuss what makes a query have poor quality results, but if it was that easy, I wouldn’t be writing a blog post about it, now would I?

The first step is understanding the value of the query triage meeting. The purpose of the meeting is to look at poor performing queries, either from logs, from human evaluation, from customer complaints, or from the participants’ own experience using your product (you are eating your own dog food, right?). Taking these inputs, you look — as a group — at what caused the results to look the way they do, what was poor about them, and how they can be improved. There should be a lively discussion of the who, what, when, where, why, and how of each query. There are a few purposes of this meeting:

1. Getting people to understand and engage with customer problems
2. Getting people to understand the search system
3. Creating a shared vocabulary around problems and solutions
4. Focusing technological solutions on those problems
5. Understanding the size and scope of problems
6. Understanding effort and investment for solutions

Like all triage the value is in focusing limited resources on the most solvable, impactful problems. Unlike other triage, this meeting is focused on bringing together different disciplines within the search community, finding problems, and prioritizing solutions.

Query Triage: How important is this (class of) relevance problem?

Here’s how that works:

1. We bring in people from multiple disciplines that have an interest in search — PM, engineering, data science, UX, CX, etc
2. We have a theme for queries for the week
3. Each person in the meeting has a query they will investigate. They bring their own or the meeting lead provides them one
4. Each participant investigates what’s wrong with the query results, how the results happened, gives it some form of classification (a vocabulary will emerge over time), and a possible solution
5. Everyone discusses their query and what they found out. Often some people will have more insight on the customer side or the engineering side or the metrics side. That’s what we want! Everyone should be discussing everyone else’s query and helping to fill in the gaps.

As time goes on, you’ll be creating a list of problem query classes that can be addressed and some methods that might address those queries. You can also review those queries over time and ask “did we fix it?”. Checking in to see if things have improved is an important step.

There are a few things that will be necessary to make all this happen:

1. A facilitator. The meeting lead will keep everything running and ensure that queries are being evaluated and everyone is participating. He or she will also be the person recording queries, problems, and solutions
2. A list of queries. This is the fun part, to me. Finding good “bad” queries can be a challenge. Use query logs with low-engagement queries, also use human evaluation to find low-relevance queries. Customer experience, customer support, survey feedback are all great sources of queries. Of course, the best source of queries to triage might come from employees who are using the service and running into problems.
3. An interdisciplinary cast of characters. Query triage works because it brings together multiple disciplines. Not everyone is going to be an expert at understanding users, understanding relevance, understanding how the search engine works, understanding metrics, etc. So, it’s important that we have many different kinds of people that can add their perspective as each query is evaluated. Over time, people will get more knowledgable about the other disciplines. That’s the beauty of query triage — your organization gets better as the search engine gets better!
4. Passion. Everyone on the team needs to be passionate about search and about creating a great user experience. If they aren’t … well they probably shouldn’t be working on your search team.

In the end, what’s the value? We bring people together to discuss bad queries, what’s the big deal? The big deal is that we use the query triage meeting as a space to discuss the problems our search engine faces, categorize them, and prioritize solutions. Not every problem is equally important, not every solution equally valuable. This approach merges the human and statistical approaches to search. It gives us a tool to discuss what the problems are and what technology is available to solve those problems. Without that understanding, you will have problems in search of solutions and solutions in search of problems.

An example I mentioned in my first post was using price as a relevance signal within certain categories. If you know that relevance of searches in the “appliances” category is poor, and you know the problem is parts & accessories pushing out actual refrigerators, then you can create a solution that boosts articles in the relevant price points. Now, all you need is understanding if that boost is feasible and to figure out the right price points. Having the right people in the room greatly facilitates that type of conversation. You can immediately understand if it is feasible and start looking at methods to determine price points. No additional meetings, just mainlining ideas!

The value, then, is that we are bringing together the technical and human and finding the intersection. What are the places where we can have the most impact for the least investment? In a word, it’s “discovery.” In two words, it’s “customer obsession” and a way to spread that obsession to all the members of the team. If there’s anything that drives success in technology, it’s customer obsession. Get focused on your customers and figure out how you can do what you do better for them, even in a technical arena like search … especially in an arena like search, perhaps.

Use a query triage meeting to understand your customers, their problems, and the technology you have available to solve them. That’s the secret weapon to merge your you human-centered and statistical approaches to search relevance.

In case you were wondering if Google has these meetings. Google has these meetings :)