To determine how correlated two metrics are to each other, they add their popularity and absolute scores, enabling them to better understand their relationship. The two basic approaches to calculating these scores are:
- Popularity score: The more frequently two metrics are selected together across experiments, the higher the score assigned to their relationship. They use the Jaccard Index to help users discover the most relevant metric once they select their initial metric. This score accounts for the experimenters’ metrics selection from past experiments.
- Absolute score: Using their XP, they can generate a pool of user samples from their metrics and calculate the Pearson correlation score of the two metrics. This accounts for serendipitous discovery; namely, the experimenter may not have considered adding a metric to the experiment since it is not directly related, but it might be moving with the user-selected metric.
After calculating these two scores, they add the score of the two steps above with relative weights on each term and recommend the metrics with the highest score to the experimenter based on their first choice of metrics.