We donate 60% of ad revenue towards data for learners who need it the most.
First time here? Checkout the FAQs!
x
Institutions: Global |ALU | TUT | UCT | UJ | UNISA | UZ | Wits

+1 vote
66 views
What continuous experiments does the Uber experimentation team perform?
in Data Science by Wooden (2,570 points) | 66 views

1 Answer

+1 vote
Best answer

To accelerate innovation and learning, the data science team at Uber is always looking to optimize driver, rider, eater, restaurant, and delivery-partner experiences through continuous experiments. Their team has implemented bandit and optimization-focused reinforcement learning methods to learn iteratively and rapidly from the continuous evaluation of related metric performance.

Recently, they completed an experiment using bandit techniques for content optimization to improve customer engagement. The technique helped improve customer engagement compared to classic hypothesis testing methods. The diagram belowoutlines Uber’s various continuous experiment use cases, including content optimization, hyper-parameter tuning, spend optimization, and automated feature rollouts:  

In Case Study 1, they outline how bandits have helped optimize email campaigns and enhance  rider engagement at Uber. Here, the Uber Eats Customer Relationship Management (CRM) team in Europe, the Middle East, and Africa (EMEA) launched an email campaign to encourage order momentum early in the customer life cycle. The experimenters plan to run a campaign with ten different email subject lines and find out the best subject line in terms of the open rate and the number of open emails. The diagram below details this case study:

 

A second example of how they leverage continuous experiments is parameter tuning. Unlike the first case, the second case study uses a more advanced bandit algorithm, the contextual multi-armed bandit technique, which combines statistical experiments and machine learning modeling. They use contextual MAB to choose the best parameters in a machine learning model.

As depicted below, the Uber Eats Data Science team leveraged MAB testing to create a linear programming model, called the multiple-objective optimization (MOO), that ranks restaurants on the main feed of the Uber Eats app:

 

The algorithm behind MOO incorporates several metrics, such as session conversion rate, gross booking fee, and user retention rate. However, the mathematical solution contains a set of parameters that they need to give to the algorithm.

These experiments contain many parameter candidates for use with their ranking algorithms. The ranking results depend on the hyper-parameters they chose for the MOO model. Therefore, to improve the performance of the MOO model, they hope to figure out the best hyper-parameters via multi-armed bandits algorithm. The traditional A/B test framework is too time-intensive to handle each test, so they decided to utilize the MAB method for these experiments. MAB is able to provide a framework to quickly tune these parameters.

They chose the contextual MAB and the Bayesian optimization methods to find the maximizers of a black box function optimization problem. Figure below, outlines the setup of this experiment:  

As shown above, contextual Bayesian optimization works well with both personalized information and exploration-exploitation trade-offs.

 

 

by Wooden (2,570 points)

Welcome to MathsGee Answers, Africa’s largest free personalized study network that helps people find answers to homework problems and connect with experts for improved outcomes.


*Inspired by Ecosia, MathsGee pledges that 60% of the revenue from the displayed adverts will be used to buy data for South African learners who need it the most.


Enter your email address:

Wellness For Entrepreneurs

Episode 24: How to be a great thinker in business – metacognition
Posted on Thursday March 04, 2021

Have you ever admired how classical great entrepreneurs like Elon Musk, or Bill Gates are able to think about various issues in a way that is above average? Metacognition – or thinking about...

Click Here To Read More.
 

Congratulations to our winners!
Posted on Monday March 01, 2021

Congratulations to Lukhanyo Maneli and Terence Molepo for winning copies of the Nuts and Bolts book written by Dr. McLean Sibanda! We cannot wait for you to dig in and join us in the book review...

Click Here To Read More.
 

Episode 23: How to lead, manage strategy, and execute, with Dr. Alex Granger
Posted on Thursday February 25, 2021

Entrepreneurs are often building the plane, cleaning it, and flying it all at once. They must plan and execute their strategies all at the same time. How do they become excellent at leading...

Click Here To Read More.
 

Special Episode: Nuts and Bolts Book Review
Posted on Sunday February 21, 2021

We are doing a book review and you could be a winner! Dr. McLean Sibanda is the author of Nuts and Bolts – a book about strengthening the innovation and entrepreneurship ecosystems of Africa....

Click Here To Read More.
 

Episode 22: How to spot new opportunities, with Kwame Bekoe
Posted on Thursday February 18, 2021

Entrepreneurs discover or create opportunities that ultimately lead to some of our favourite products and solutions to real pressing problems. But how exactly do they spot these opportunities in...

Click Here To Read More.
 

|STEM Gender Equality | ZOOM | Slack | eBook