Variance Reduction for Experiments in Low-Frequency E-Commerce Context
I'm a UX researcher and have been reading up on CUPED for an upcoming experiment. I understand that CUPED is currently set up in Optimizely Analytics to work on the past two weeks of data for the metric of interest, which must be numeric (e.g., Revenue). As is the case for many low-frequency e-commerce products, many of our customers won't have revenue events in the two weeks leading up to an experiment.
As I dove deeper down the rabbit hole, I came across CUPAC, developed by DoorDash in 2020. CUPAC solves the above problem by using a model-based predicted outcome as the covariate, computed from features observable at exposure time. DoorDash's own benchmark shows 10-20% sample size reduction, which compounds the value Optimizely already delivers. I'd love to have CUPAC as part of Optimizely Analytics.