Izlaganje sa skupa
https://doi.org/10.2478/zireb-2018-0004
Are Multi-Armed Bandits Susceptible to Peeking?
Markus Loecher
; Berlin School of Economics and Law, Berlin, Germany
Sažetak
A standard method to evaluate new features and changes to e.g. Web sites is A/B testing. A common pitfall in performing A/B testing is the habit of looking at a test while it's running, then stopping early. Due to the implicit multiple testing, the p-value is no longer trustworthy and usually too small. We investigate the claim that Bayesian methods, unlike frequentist tests, are immune to this "peeking" problem. We demonstrate that two regularly used measures, namely posterior probability and value remaining are severely affected by repeated testing. We further show a strong dependence on the prior probability of the parameters of interest.
Ključne riječi
multiple comparisons; A/B testing; Bayesian decision theory
Hrčak ID:
200885
URI
Datum izdavanja:
31.5.2018.
Posjeta: 1.424 *