We donate 60% of ad revenue towards data for learners who need it the most.
First time here? Checkout the FAQs!
Institutions: Global |ALU | TUT | UCT | UJ | UNISA | UZ | Wits

0 votes
How does sequential testing offer a way to continuously monitor key business metrics at Uber?
in Data Science by Wooden (2,570 points) | 21 views

1 Answer

0 votes
Best answer

While traditional A/B testing methods (for example, a t-test) inflate Type-I error by repeatedly taking subsamples, sequential testing offers a way to continuously monitor key business metrics.

One use case where a sequential test comes in handy for the Uber experimentation team is when identifying outages caused by the experiments running on their platform. They cannot wait until a traditional A/B test collects sufficient sample sizes to determine the cause of an outage; they want to make sure experiments are not introducing key degradations of business metrics as soon as possible, in this case, during the experimentation period. Therefore, they built a monitoring system powered by a sequential testing algorithm to adjust the confidence intervals accordingly without inflating Type-I error.

Using their XP, they conduct periodic comparisons about these business metrics, such as app crash rates and trip frequency rates, between treatment and control groups for ongoing experiments. Experiments continue if there are no significant degradations, otherwise they will be given an alert or even paused. The workflow for this monitoring system is shown below:




by Wooden (2,570 points)

Related questions

0 votes
0 answers
asked Jul 28, 2020 in Data Science by Tedsf Diamond (55,170 points) | 18 views
0 votes
1 answer
+1 vote
1 answer
0 votes
1 answer

Welcome to MathsGee Answers, Africa’s largest free personalized study network that helps people find answers to homework problems and connect with experts for improved outcomes.

*Inspired by Ecosia, MathsGee pledges that 60% of the revenue from the displayed adverts will be used to buy data for South African learners who need it the most.

Enter your email address:

Wellness For Entrepreneurs

Episode 24: How to be a great thinker in business – metacognition
Posted on Thursday March 04, 2021

Have you ever admired how classical great entrepreneurs like Elon Musk, or Bill Gates are able to think about various issues in a way that is above average? Metacognition – or thinking about...

Click Here To Read More.

Congratulations to our winners!
Posted on Monday March 01, 2021

Congratulations to Lukhanyo Maneli and Terence Molepo for winning copies of the Nuts and Bolts book written by Dr. McLean Sibanda! We cannot wait for you to dig in and join us in the book review...

Click Here To Read More.

Episode 23: How to lead, manage strategy, and execute, with Dr. Alex Granger
Posted on Thursday February 25, 2021

Entrepreneurs are often building the plane, cleaning it, and flying it all at once. They must plan and execute their strategies all at the same time. How do they become excellent at leading...

Click Here To Read More.

Special Episode: Nuts and Bolts Book Review
Posted on Sunday February 21, 2021

We are doing a book review and you could be a winner! Dr. McLean Sibanda is the author of Nuts and Bolts – a book about strengthening the innovation and entrepreneurship ecosystems of Africa....

Click Here To Read More.

Episode 22: How to spot new opportunities, with Kwame Bekoe
Posted on Thursday February 18, 2021

Entrepreneurs discover or create opportunities that ultimately lead to some of our favourite products and solutions to real pressing problems. But how exactly do they spot these opportunities in...

Click Here To Read More.

|STEM Gender Equality | ZOOM | Slack | eBook