Theory and practice of large scale online causal estimation

The growth of online technology companies heavily relies on their ability to innovate and improve their products.
To rigorously measure the positive impact of new interventions, such as newly introduced features, and enable effective data-driven innovation, practitioners routinely rely on A/B tests (or online randomized experiments). In this talk, we will discuss the main challenges and open research questions faced by teams running large-scale experiments in online A/B testing.
After outlining the classic model for online experiments, we will show how recent advances in the field have allowed to improve over this classical approach. In particular, we will discuss how modeling improvements (e.g., Bayesian inference and regression models leveraging covariate adjustments), as well as predictions methods aimed to determine the optimal duration of an experiment have impacted the theory and practice of online experimentation.
We will conclude our discussion with open research questions and challenges, with an emphasis of limitation of experimentation under interference, a common issue in online experimentation.