Google Analytics - Multi-armed Bandit Experiments

Multi-armed Bandit Experiments - no wonder my A-B stats look like they do...

Steven L. Scott, PhD, Sr. Economic Analyst, Google
January 23, 2013 | 9:40 AM

---

This article describes the statistical engine behind Google Analytics Content Experiments. Google Analytics uses a multi-armed bandit approach to managing online experiments. A multi-armed bandit is a type of experiment where:

• The goal is to find the best or most profitable action
• The randomization distribution can be updated as the experiment progresses

The name "multi-armed bandit" describes a hypothetical experiment where you face several slot machines ("one-armed bandits") with potentially different expected payouts. You want to find the slot machine with the best payout rate, but you also want to maximize your winnings. The fundamental tension is between "exploiting" arms that have performed well in the past and "exploring" new or seemingly inferior arms in case they might perform even better. There are highly developed mathematical models for managing the bandit problem, which we use in Google Analytics content experiments.

Read More