Get Help And Discuss STEM Concepts From Math To Data Science & Financial Literacy
STEM Gender Equality | Join us on ZOOM | Spreading Mathematical Happiness

MathsGee is free of annoying ads. We want to keep it like this. You can help with your DONATION

0 like 0 dislike
16 views
Why are A/B test experiments impossible at times?
in Data Science by Diamond (50,545 points) | 16 views

1 Answer

0 like 0 dislike

Sometimes it’s just not possible to set up an experiment. Here are a few reasons why A/B tests won’t work in every situation:

  • Lack of tooling. For example, if your code can’t be modified in certain parts of the product.
  • Lack of time to implement the experiment.
  • Ethical concerns  for example, at an e-commerce company like Shopify, randomly leaving some merchants out of a new feature that could help them with their business is sometimes not an option).
  • Just plain oversight (for example, a request to study the data from a launch that happened in the past).

Fortunately, if you find yourself in one of the above situations, there are methods that exist  which enable you to obtain causal estimates.

A quasi-experiment  is an experiment where your treatment and control group are divided by a natural process that isn’t truly random, but are considered close enough to compute estimates. Quasi-experiments frequently occur in product companies, for example, when a feature rollout happens at different dates in different countries, or if eligibility for a new feature is dependent on the behaviour of other features (like in the case of a deprecation). In order to compute causal estimates when the control group is divided using a non-random criterion, you’ll use different methods that correspond to different assumptions on how “close” you are to the random situation.

by Diamond (50,545 points)

Related questions

0 like 0 dislike
1 answer
asked Oct 18 in Data Science by Tedsf Diamond (50,545 points) | 6 views
0 like 0 dislike
1 answer
0 like 0 dislike
0 answers
0 like 0 dislike
1 answer

Welcome to MathsGee Q&A Bank, Africa’s largest personalized STEM and Financial Literacy education network that helps people find answers to problems, connect with others and take action to improve their outcomes.


MathsGee Q&A is the STEM and Financial Literacy knowledge-sharing community where students and experts put their heads together to crack their toughest questions.


MathsGee is free of annoying ads. We want to keep it like this. You can help with your DONATION

Enter your email address:

11,560 questions
9,542 answers
100 comments
10,443 users