Overview

When results are shown for a search query, we want to show the most relevant products on the top.

For capturing the relevance, one way is to find which products are being clicked by the users. In the best scenario, the users will be clicking the top product.

For example, when we searched for "Men Sport Shoe", it showed following list of products. The results will be shown to multiple customers who searched with the same query.

For measuring the effectiveness of the ‘relevance’ of the result, we can find where customers are clicking and if it is perfect scenario all the customers will be clicking on the first product.

As customers click on the product down the list, we consider it as an improvement area for the product ranking. The place of products being clicked is measured using Reciprocal Rank and this is a commonly used metric for measuring Ranking Performance.

RR= 1/position of first click on a product

Mean Reciprocal Rank for a Search Query

Now, there can be multiple customers who search with same query, and we want to measure effectiveness of the results for the search query, and we take average of Reciprocal Rank (RR) across customers for the same search query. For example, we assume 4 customers have used search query "Men Sport Shoe" and customers have clicked on the products with different positions. First customer clicked on a product at position 2, so the Reciprocal Rank is ½, second customer we assume that same set of products are fetched found the top product as the most relevant (RR is 1/1).

MRR for search query term based on these 4 users is

MRR = ¼ (1/2 + 1/1+1/7+1/4)

We need to calculate MRR for each query to track and monitor MRR for each of the queries separately so that we can find best and worst queries.

MMR For a Model

Machine Learning Model can help us in improving relevance ranking for the search queries and across users on the ecommerce platform.  For improving a model, we need to measure the performance on current model (champion) and compare performance of a competing model (challenger).

 Any model can perform well on a few search queries, so we need to calculate MMR across all the search queries. We take average of MMR across.

MMR (Model) = Average of MMR queries

We need to take average of Reciprocal Ranks across users and queries. If there are Q total queries and rank for each query is Ri, then it can be depicted as

MRR = (1/Q )

If all the query results are perfect and all the users click on the top products, the rank for each query will be 1 and if we take average, it will be 1. So, best MMR value will be 1.

Assume for a scenario, we show max 40 products and if users find last products as most relevant, hence rank will be 1/40 for each query. Now we need to take average across queries (Q) and it will be (1/Q) (Q/40) = 1/40

Challenges

• Reciprocal rank for each of the search query and user is considered based on the first click of a user only. 
• It does not give credit for pulling multiple relevant products
• User may click on a product, go to detail, comes back, and click on the other product from the list. These details are not typically considered in the MRR calculation.
• If a user does not find the results relevant, the user may leave without any click, and this effect is not getting captured in MRR.  

References

https://en.wikipedia.org/wiki/Mean_reciprocal_rank