Lyle's idea for upgrade supporting better-informed incremental feature development, essentially upgrade experiments where a user can include or exclude themselves. I think "self enrollment" could be used as the technical term?
As I understood it, this should be fairly straightforward, but of course i've thought up some complications:
Requirements:
- A researcher can create an experiment that allows participants to add/remove themselves to an include list
- A researcher can create an experiment that allows participants to add/remove themselves to an exclude list
- A user can exclude themselves from an experiment
- A user can include themselves in an experiment
Technically speaking:
-
An experiment would need to accommodate a sort of intermediary inclusion state, where their "assignment" can be simply "selfEnrollmentEligible" for that experiment, no actual condition sent yet. (what will "/mark" do, enroll them in "eligible"? so we can measure metrics of the group that has seen the option and ignored it?)
-
A new client sdk endpoint separate from /mark to send a signal to upgrade that a user is choosing their enrollment, such as /voluntary-enrollment, where we can post an request like { <experiment-uuid>: true } (User-Id is already in the header).
-
If that experiment and student is valid as 'selfEnrollmentEligible', they will then do the equivalent of marking that condition, and the response can be the actual condition assignment. That apparently would mean switching conditions, which i do think would work, but is a novel thing to try.
Follow-up questions:
What states do we allow an experiment to be open to voluntary-enrollment? Do we allow in paused/stopped?
What are the implications to switching conditions, data-wise?
What does it mean to say opt-out? If a student excludes themselves, does it mean they don't want to try the feature but we can still measure them as a set of users who we are collecting data on? As-in, do they get excluded, or essentially allow this to mean they "enroll" by setting themselves to a self-selected control condition or something....?
Lyle's idea for upgrade supporting better-informed incremental feature development, essentially upgrade experiments where a user can include or exclude themselves. I think "self enrollment" could be used as the technical term?
As I understood it, this should be fairly straightforward, but of course i've thought up some complications:
Requirements:
Technically speaking:
An experiment would need to accommodate a sort of intermediary inclusion state, where their "assignment" can be simply "selfEnrollmentEligible" for that experiment, no actual condition sent yet. (what will "/mark" do, enroll them in "eligible"? so we can measure metrics of the group that has seen the option and ignored it?)
A new client sdk endpoint separate from /mark to send a signal to upgrade that a user is choosing their enrollment, such as
/voluntary-enrollment, where we can post an request like{ <experiment-uuid>: true }(User-Id is already in the header).If that experiment and student is valid as 'selfEnrollmentEligible', they will then do the equivalent of marking that condition, and the response can be the actual condition assignment. That apparently would mean switching conditions, which i do think would work, but is a novel thing to try.
Follow-up questions:
What states do we allow an experiment to be open to voluntary-enrollment? Do we allow in paused/stopped?
What are the implications to switching conditions, data-wise?
What does it mean to say opt-out? If a student excludes themselves, does it mean they don't want to try the feature but we can still measure them as a set of users who we are collecting data on? As-in, do they get excluded, or essentially allow this to mean they "enroll" by setting themselves to a self-selected
controlcondition or something....?