Above is a chart outlining the types of experimentation methodologies that the Uber Experimentation Platform team uses:
There are various factors that determine which statistics methodology Uber should apply to a given use case. Broadly, they use four types of statistical methodologies:
- fixed horizon A/B/N tests (t-test, chi-squared, and rank-sum tests),
- sequential probability ratio tests (SPRT),
- causal inference tests (synthetic control and diff-in-diff tests), and
- continuous A/B/N tests using bandit algorithms (Thompson sampling, upper confidence bounds, and Bayesian optimization with contextual multi-armed-bandit tests, to name a few).
They also apply block bootstrap and delta methods to estimate standard errors, as well as regression-based methods to measure bias correction when calculating the probability of type I and type II errors in their statistical analyses.