What does it mean to be a Product Manager (PM)? What does the search team do? What does the PM do for the search team at your organization? Is there a difference between Search PMs and other PMs?

First, let’s talk about the PM role in tech, more broadly.

The Product Manager

Product management might be the most misunderstood discipline in tech. If I were being glib, I could sum it up with scene from the movie “Office Space” where the two Bobs ask Tom “What would you say … you do here?” …

We’ve talked about statistical vs human approaches to search, we’ve talked about metrics, and we’ve talked about A/B testing. Now we’re going to bring it all together in something I like to call “the Launch Review”

Image for post
Image for post
Even rocket scientists do launch reviews.

The Launch Review is simple; it’s the meeting where we decide to launch a search algorithm (or any other experiment, really). It’s also complex; there are many different priorities, stakeholders, and agenda that must be considered when deciding to launch. Experiments don’t happen in a vacuum, and that’s especially true for search experiments that can majorly impact the user experience of your product/app/website.


The great advantage of the internet is that it allows anyone to publish anything (shout out to Medium!). The old gatekeepers are gone, so previously marginalized views can proliferate. This is wonderful.

The great disadvantage of the internet is that it allows anyone to publish anything. The old gatekeepers are gone, so previously marginalized views can proliferate. This is terrible.

The truth is that the internet is wonderful because of the low barrier to entry, it allows people who are previously un-heard-from to share their stories. This is wonderful. It also allows nuts, trolls, or bots to spread misinformation at…

In our last installment, we learned about the important of metrics and how the choice of metrics is so important to building, maintaining, and improving our search system. In today’s episode, we’ll be discussing where the rubber meets the road of search relevance improvement: experimentation (aka A/B testing, aka split testing, etc)

Experimentation, in my mind, is what separates the Data Scientist from a Machine Learning Engineer. A brief primer on experimentation: experimentation is a controlled method of testing a hypothesis. A hypothesis is an educated guess, sure, but it’s also a guess that can be measured. It’s your line…

What gets measured gets managed - Peter Drucker

We’ve all heard the aphorism “What gets measured gets managed.” It’s a truism in today’s “data driven” culture. But what exactly are you measuring, and what are you managing when you measure it? With search, it’s not as straightforward as you might think. Without a clear conception of what goes into a search metric, you may find yourself optimizing for the wrong user experience. Read on to see how different metrics can optimize for different things in your search experience.

First, a (very) brief primer on how search engines work. Search engines…

Remember way back in my first post on search relevance how I said there’s a statistical and a human-centered approach to search improvement? Well this post is about one way you can merge your human and metrics-driven approaches to maximize the value of each. This secret weapon is called a Query Triage.

The first thing to understand is that while there are two distinct approaches to search relevance improvement, but neither really lives in a bubble… unless your organization is dysfunctional. …

In my first couple of posts, I described the need for human- and metrics-centered approaches to relevance improvement and how we can evaluate relevance in online and offline approaches. This post will focus on the offline approach to relevance evaluation and describe how to set up a human judgment program to effectively evaluate search relevance*.

Image for post
Image for post

As I mentioned in the previous post, human-rated relevance judgements are critical to measuring (and therefore improving) search relevance. However, many search teams don’t know where to start when it comes to creating a program of relevance evaluation.

The first step in establishing a relevance…

In my first post on search I discussed two strategies for human or metrics-centered approaches to improving search. Now, I’m going to introduce you to a concept that is a bit of both: how to measure search.

There are two main ways to measure search relevance: online (log-based) and offline (human-rated). There are distinct advantages and disadvantages to each, but it’s important to know that these approaches complement each other. It’s not one-or-the other, to really measure search (and improve it) you need both approaches!

When it comes to improving our products in a data-driven way, A/B testing is considered…

For search product teams, search is like baseball, not golf.

In golf, you get a little closer to the hole with every stroke (unless you are as bad at golf as I am)

search is like baseball, the more chances we take the more likely we’ll be successful
search is like baseball, the more chances we take the more likely we’ll be successful
Photo by Chris Chow on Unsplash

In baseball, you come up to the plate never knowing if it will be a hit, a walk, a strikeout, or a home run. So, the goal in search is to get as many “at-bats” as possible. The more chances you take, the more opportunities you’ll have for that home run. For search teams, working on search relevance improvement is the same. We only get so many…

I recently hired a data scientist for my team at work. This is very exciting. However, it took 6 months to do it. I want to explore why I had such a difficult time finding someone to be a data scientist, through the lens of what it means to be a data scientist.

What is a data scientist? A data scientist is someone who uses data to answer questions using the scientific method. The scientific method, in case you have forgotten 5th grade goes like this:

Image for post
Image for post

According to my friend, Dan, a good data scientist needs a good problem https://medium.com/@dfrankow/how-to-become-a-data-scientist-part-1-find-a-good-problem-2971442227cc

James Rubinstein

Search nerd, data nerd, and all-around nerd-nerd. He has worked at eBay, Apple, and Pinterest, and currently leads the Product Analytics team at LexisNexis

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store