qa mode versus preview
AS an experimentation builder
I WOULD LIKE an easy way to fully qa test and share pre-production experiments
BECAUSE i want to validate all the aspects of my experiment and be able to share pre-production model with non-technical partners
FUNCTIONALITY
look for a cookie or querystring parm or hostname and from that set 'qa' to either true or false.
false: take no action
true:
parms: xCode, info, remove
xCode sets qa cookie and allows you into the experiment you match the qa mode audience for
info allows for specifically coded console.logs to run
remove - removes qa cookie / environment to rest user to normal
all lower environments automatically set to qa which turns on variation logging
what this mode enables
sets all custom events to "environment=qa"
allows user to enter an experiment that has a qa mode audience.
if you set your cookie to a specific experiment - provided that the experiment was set up properly with qa mode audience and is published.
provides special gui console.logs for the experiment code.
when entering from querystring, sets qa cookie so visitor stays in qa mode until they reset with 'remove'
this basically puts a system around cookie based qa and allows you to send out links to partners with "optiQa=x###" in the querystring to review.
IMPLEMENTATION:
besides having publish and preview in the top left of the nav, could have a toggle/radio that said 'qa -- live' that sets the base for this and automates all of what I am doing above. when turning it on, you'd get an id of sorts that would work for the parm/cookie. No audience addition or wrangling. could add the logging as a util - util.log('comment'); or something like that. this could also serve as the 'start date' for metrics - when it is turned to 'live'. The codes should be as short as possible - just incrementing integers starting at 1 to be easy to use.
WHY
i want to qa experiments that essentially run in the same manner as they do for an actual visitor - instead of previewing or variation forcing because they skip audiences, the timing is different AND events don't fire.
i want to see the page activate and potential impact to js racing conditions, fouc, and timing on redirects. i want to validate the audiences i've set up. i want to ensure events are firing. basically, i want to see what my visitors see, not just what would happen in an ideal world.
lastly, is easier to qa in mobile without the preview overlay, which causes qa team issues.
-
Kristian Reich
commented
I marked this as nice to have because I already built a system in project.js to do this. It would be far easier to obscure all the pieces of it with a feature instead of having to create the audience each time and then manage those. It would also be easier for other and new experiment publishers to manage.