This feedback is on behalf of Vistaprint. - Mauricio Acebal and Lukas Vermeer.
This feedback is related to the special treatment of the standard "Off" variation.
Vistaprint are happy they are capable of turning off a variation without changing traffic allocation, but have feedback regarding the definition of how we meant the "off variation to work".
In this scenario, they have variations "A", "B", "C" and "D". After a week, they realize that variation "D" is causing issues, so they change it to "Off". then, a week later, they want to turn off another variation, variation "C". My first expectation (which has been confirmed by Josh Holt) is to delete the variation, and keep the traffic allocation the same for "A" and "B", but change the off variation to reflect the traffic allocation from "C" and "D" together. Is this the right assumption? In their mind, they see the "off variation" as a solution to this, effectively having the option to turn multiple variations "off".
They see our solution as potentially problematic, because it could introduce bias in the results for two reasons:
Traffic from before the change would be underrepresented in the overall results.
Returning traffic that was previously exposed to the stopped treatment might return and be re-allocated to a different variation, causing spillover effects.
Could we have a conversation internally on this topic, to see if it makes sense for us to allow customers to use the "off" variation multiple times, to avoid these biases?
