Informed Bayesian Inference for the A/B Test

2021
Booming in business and a staple analysis in medical trials, the A/B test assesses the effect of an intervention or treatment by comparing its success rate with that of a control condition. Across many practical applications, it is desirable that (1) evidence can be obtained in favor of the null hypothesis that the treatment is ineffective; (2) evidence can be monitored as the data accumulate; (3) expert prior knowledge can be taken into account. Most existing approaches do not fulfill these desiderata. Here we describe a Bayesian A/B procedure based on Kass and Vaidyanathan (1992) that allows one to monitor the evidence for the hypotheses that the treatment has either a positive effect, a negative effect, or, crucially, no effect. Furthermore, this approach enables one to incorporate expert knowledge about the relative prior plausibility of the rival hypotheses and about the expected size of the effect, given that it is non-zero. To facilitate the wider adoption of this Bayesian procedure we developed the abtest package in R. We illustrate the package options and the associated statistical results with a fictitious business example and a real data medical example.
JOURNAL OF STATISTICAL SOFTWARE
页码:1-39|卷号:100|期号:17
ISSN:1548-7660
收录类型
SSCI
发表日期
2021
学科领域
循证社会科学-方法
国家
荷兰
语种
英语
DOI
10.18637/jss.v100.i17
其他关键词
R PACKAGE; SELECTION; EQUALITY; RATIOS
资助机构
Netherlands Organization for Scientific Research (NWO)Netherlands Organization for Scientific Research (NWO) [406.16.528]; NWO Vici grantNetherlands Organization for Scientific Research (NWO) [016.Vici.170.083]
资助信息
This research was supported by a Netherlands Organization for Scientific Research (NWO) grant to QFG (406.16.528) and by an NWO Vici grant to EJW (016.Vici.170.083).
被引频次(WOS)
2
被引更新日期
2022-01
来源机构
University of Amsterdam University of Amsterdam
关键词
model comparison Bayes factor prior elicitation Bayesian estimation