We love feedback from you on our products and the problems in your daily work that you would like us to solve. Please describe the challenge you're encountering and your desired outcome. Be as detailed as possible.
For technical issues or bugs please head to Support or our Developer Community. You can assign up to 20 votes in total. Thank you for your feedback.
Status explanation: 'Future Consideration' = Continuing to collect further feedback, not planned at this time. 'Investigating' = Prioritized for deeper customer and feasibility investigations ahead of planning development.
This feedback is on behalf of Vistaprint. - Mauricio Acebal and Lukas Vermeer.
This feedback is related to the special treatment of the standard "Off" variation.
Vistaprint are happy they are capable of turning off a variation without changing traffic allocation, but have feedback regarding the definition of how we meant the "off variation to work".
In this scenario, they have variations "A", "B", "C" and "D". After a week, they realize that variation "D" is causing issues, so they change it to "Off". then, a week later, they want to turn off another variation, variation "C". My first expectation (which has been confirmed by Josh Holt) is to delete the variation, and keep the traffic allocation the same for "A" and "B", but change the off variation to reflect the traffic allocation from "C" and "D" together. Is this the right assumption? In their mind, they see the "off variation" as a solution to this, effectively having the option to turn multiple variations "off".
They see our solution as potentially problematic, because it could introduce bias in the results for two reasons:
Traffic from before the change would be underrepresented in the overall results.
Returning traffic that was previously exposed to the stopped treatment might return and be re-allocated to a different variation, causing spillover effects.
Could we have a conversation internally on this topic, to see if it makes sense for us to allow customers to use the "off" variation multiple times, to avoid these biases?