We donate 60% of ad revenue towards data for learners who need it the most.
First time here? Checkout the FAQs!
Institutions: Global |ALU | TUT | UCT | UJ | UNISA | UZ | Wits

+1 vote
What methodologies does Uber employ to perform sequential testing for metrics monitoring purposes?
in Data Science by Wooden (2,570 points) | 231 views

1 Answer

+1 vote
Best answer

The Uber experimentation team leverages two main methodologies to perform sequential testing for metrics monitoring purposes: the mixture sequential probability ratio test (mSPRT) and variance estimation with FDR.

Mixture Sequential Probability Ratio Test

The most common method they use for monitoring is mSPRT. This test builds on the likelihood ratio test by incorporating an extra specification of mixing distribution H. Suppose we are testing the metric difference with the null hypothesis being $\theta$ , then the test statistics could be written as:

Since we have large sample sizes and the central limit theorem can be applied to most cases, we use normal distribution as our mixing distribution, $H ~ N(0,r^2)$. This leads to easy computation and a closed form expression for  

Another useful property about this method is under null hypothesis, nH, 0 is proven to be a martingale:

 . Following this, we could construct $(1 - \alpha)$ confidence interval.


Variance estimation with FDR control

To apply sequential testing correctly, we need to estimate variance as accurately as possible.  Since we monitor the cumulative difference between our control and treatment groups on a daily basis, observations from the same users introduce correlations which violate the assumption of the mSPRT test. For example, if we are monitoring click through rates, then the metric from one user across multiple days may be correlated. To overcome this, we use delete-a-group jackknife variance estimation/block bootstrap methods to generalize mSPRT test under correlated data.

Since our monitoring system wants to evaluate the overall health of an ongoing experiment, we monitor many business metrics at the same time, potentially leading to false alarms. In theory, either the Bonferroni or BH correction could be applied in this scenario. However, since the potential loss of missing business degradations can be substantial, we apply BH correction here and also tune in parameters (MDE, power, tolerance for practical significance, etc.) for metrics with varying levels of importance and sensitivity.

by Wooden (2,570 points)
edited by

Welcome to MathsGee Answers, Africa’s largest free personalized study network that helps people find answers to homework problems and connect with experts for improved outcomes.

*Inspired by Ecosia, MathsGee pledges that 60% of the revenue from the displayed adverts will be used to buy data for South African learners who need it the most.

Enter your email address:

Wellness For Entrepreneurs

Episode 24: How to be a great thinker in business – metacognition
Posted on Thursday March 04, 2021

Have you ever admired how classical great entrepreneurs like Elon Musk, or Bill Gates are able to think about various issues in a way that is above average? Metacognition – or thinking about...

Click Here To Read More.

Congratulations to our winners!
Posted on Monday March 01, 2021

Congratulations to Lukhanyo Maneli and Terence Molepo for winning copies of the Nuts and Bolts book written by Dr. McLean Sibanda! We cannot wait for you to dig in and join us in the book review...

Click Here To Read More.

Episode 23: How to lead, manage strategy, and execute, with Dr. Alex Granger
Posted on Thursday February 25, 2021

Entrepreneurs are often building the plane, cleaning it, and flying it all at once. They must plan and execute their strategies all at the same time. How do they become excellent at leading...

Click Here To Read More.

Special Episode: Nuts and Bolts Book Review
Posted on Sunday February 21, 2021

We are doing a book review and you could be a winner! Dr. McLean Sibanda is the author of Nuts and Bolts – a book about strengthening the innovation and entrepreneurship ecosystems of Africa....

Click Here To Read More.

Episode 22: How to spot new opportunities, with Kwame Bekoe
Posted on Thursday February 18, 2021

Entrepreneurs discover or create opportunities that ultimately lead to some of our favourite products and solutions to real pressing problems. But how exactly do they spot these opportunities in...

Click Here To Read More.

|STEM Gender Equality | ZOOM | Slack | eBook