Reddit doesn’t (at least as far as I know) store a history of edits, so what is saved on the database is what your comment literally is. The reason people suggest overwriting comments is because the comment itself has value (for a variety of reasons), so overwriting the comment with something valueless (in the sense that it has no value for Reddit) is better, so the database itself is updated with that valueless comment.
After that whatever you do with the account is up to you.
I in general find lay people have a very weak understanding of how research functions. This is a very generic statement, but everything from IRB processes to how science is reported in manuscripts and everything in between tends to be a quagmire, and this is absolutely with recognition that some of this process is mired in red-tape, bureaucracy, and endless administration.
For example, there’s a long-standing idea that IRBs are the gatekeepers of research. In reality, any IRB worth their weight (and really, all of them are for compliance) should be viewed as a research stakeholder. They should be there to make research happen and let scientists do the best research they can with the minimal amount of harm to participants. Sometimes this involves compromises, or finding alternatives that are less harmful, and this is a good thing. No researcher should dislike their IRB at all, but criticizing unnecessary process & paperwork is very much a matter of process rather than a matter of what the IRB does. A IRB taking a long time to turn something around is a different matter from the IRB exhaustively reviewing the proposed work and returning questions that need to be addressed.
Another common example is scientific studies are frequently criticized about sample sizes. Yes, a lot of research would definitely benefit from better sampling and larger samples, but narrowly focusing on sample sizes misses a lot of the other considerations taken for evaluating statistical power. For example, if one wants to know whether beheading people results in injuries incongruent with life, one doesn’t really need a large sample to come to this conclusion because the effect (size) is so large. Of course more numbers help, but past a point more numbers only add to the cost of the research without measurably improving the quality of the statistical inferences made. We could instead save some of this money and repurpose it for repeating a study.
And on that note, study replication is very much a really needed thing, particularly in applied research areas where we care a lot about behavioral outcomes. We don’t have enough of this, it is not funded well enough, and by and large the general public is right in that we do need more of this. It’s not exciting to do at all. On the other hand, meta-analysis papers which pull results from a large number of papers looking at a particular topic usually give a helpful benchmark on the broad direction and general take-away conclusions of a particular topic.
In this topic about IRBs, A/B UI/UX testing for the set-up that Reddit did it, and being run out of an university setting? That’s hyperbole. I don’t like businesses doing aggressive user-focused testing without informing the user, particularly with UI/UX changes I dislike (looking at you too Twitch with your constant layout changes), but at the end of the day these kinds of testing generally don’t ever rise up to the threshold needed to be a particularly meaningful blip. Insinuating otherwise vastly mischaracterize how research is done in formal, structured settings.
Reddit violates ALL of these example rules.
No.
minors use reddit and there is no indication that reddit experiments exclude them. Minors are not prohibited on the site and there is no tracking of age other than the vague “show me NSFW results” checkbox
Strictly speaking, COPPA prevents Reddit from collecting information from users under the age of 13. While there are no explicit guarantees that a person on the site isn’t 13 or older, and also recognizing that age of majority is typically 18, then in a general literal sense yes there are minors involved, in so far as the activity discussed is research.
intervention is not brief, has lasted a week or more
A research intervention is an intervention in so far as it intersects with the participant. A drug trial that lasts for 10 weeks total but only gives doses to participants for 2 weeks (and then results monitoring during, 2-weeks post, and 4-weeks post) does not mean that the “intervention” lasts 10 weeks.
In most practical terms, running an experiment for 2-3 weeks is very common to collect sufficient data. However, the intervention itself may be quite brief (for example, a short 45-minute interview with a participant would be the “intervention” for a 2-3 week long study interviewing physicians on their concerns about organizational capacity for change).
For a repeated measures experiment, the intervention usually involves the actual experiment encounter and maybe some additional time between them.
For the case of A/B testing, usually the “intervention” in this case is the A/B test as it applies to the user at the moment, and not the entire duration of when the testing is taking place for all participants.
interventions having harm
Once again, to iterate, you are equating the inability to view the content of a website without logging into an account with such substantial emotional and psychological harm that is comparable to being verbally derided in public, for a week, shamed on a public channel, and/or comparable situations. You are not going to convince an IRB that being able to view the content of a website without logging in, then subsequently going to a different community to ask for help, and then hypothetically ruminate about the matter for weeks, is going to exceed the kind of everyday ordinary harm to qualify a risk level above minimal risk.
the subject was deceived and was definitely not informed prospectively that deception may take place, neither has agreed to it; subject was not informed even retrospectively other than some random admin suggesting they were part of an experiment after they complained online; for that matter the subject was not informed that an experiment would be taking place at all and has never agreed to anything, other than possibly in the ToS.
Deception goes beyond simply “lying” to or “not informing” the participant. Duke University gives some good considerations here:
Additionally, a waiver for utilizing deception in research has to:
Satisfying #1 and #2 in an UI/UX A/B testing regime that Reddit used here is pretty easy. You are specifically hung up on the implicit harms involved but in reality they are of no particular serious concern.
#3 is particularly interesting, because effectively what that means is you need to demonstrate deception is necessary in experimental design for the experiment to actually work. If you are seeing whether a person will interact more with a site or not if they are blocked from seeing content without logging into an account, telling them ahead of time could already bias the outcome. This is in very specific consideration that #1 and #2 are already met: being unable to view a website without logging into an account is not anything more than minimal risk. And even then, it is important to emphasize that the failure to disclose the research hypothesis to counter the demand effect is NOT deception.
The kind of UI/UX A/B testing Reddit employed in this specific instance is absolutely not equivocal to the Asch conformity studies.
To be very clear here, when YOU operate under the belief that being unable to view the content of the website, then posting about it elsewhere and potentially being ridiculed for it is sufficient of a bar to meet beyond minimal risk, then we have very different definitions of what “minimal risk” entails. Since we cannot come to consensus on this particular topic, and instead you’ve gone so far as to associate this kind of activity’s harm of equivalent to the Asch conformity studies is frankly ludicrous.
If we cannot agree on this, then so be it. However, I will repeat (with added finality): YOU running the same A/B experiment Reddit is doing on an university-sanctioned website will NOT get you run out of the university by the IRB. In a real-world scenario. you would likely discuss this experiment with an administrator or similar at your department, then maybe send an e-mail off to the IRB for any clarification, would likely have some back-and-forth, and then would ultimately receive a determination that it is exempt (at worst), or not considered human subjects research at all. I can see a few circumstances where such an effort might merit an expedited review, but to do this would involve some torturous twisting of the situation that could be easily avoided (for example, running this testing on a page that has instructions for performing the Heimlich maneuver).
The Asch conformity study experiments are absolutely not equivalent to doing A/B testing on a wide scale with regards to how users interact with a website when presented with a pop-up preventing further interaction without logging in.
Edit: I would add that, with regards to “harm”, IRBs usually don’t contend themselves excessively with imaginative processes leading to catastrophic harm. Some kinds of harm are obvious (for example, disclosing that someone has a degenerative life disease discovered incidentally during clinical sequencing), and others are very much in the ballpark of “it could happen, but if you make reasonable accommodation it’s not a big deal”, such as being concerned that someone might risk choking to death while being a participant on a blind taste test of different beverages using tiny samples. Allergies are real and participants should be informed about these risks ahead of time; potentially choking on a small sip of Coke is good to note but is not going to register on even the participant’s radar.
> But the situation linked in OP is exactly of a user who was so upset by unexpected behavior secretly thrust upon him that he had to go online to ask others for help
You will need to convince the IRB that such an outcome is more than just minimal risk. The very definition of minimal risk refers to the probability and magnitude of harm. Being unable to use a website unexpectedly or being prompted to sign up for an account before being able to view something is very much not a kind of “harm” that is greater than those ordinary encountered in daily life, no more different than accidentally spilling some creamer on a granite tabletop or realizing one forgot their keys and got locked out of their room.
We can torture the words as much as we want to lead to a particular interpretation, but by and large these are not the kind of word meanderings that IRBs tolerate.
I’ve read the rest of your comment and my primary issue is your take on it and the subsequent interpretation(s) are not how these policies and practices are implemented, interpreted, and actioned at IRBs nationally.
For example, this is how Indiana University guides researchers for making exempt determinations when reaching out to their IRB. University of South Carolina’s IRB provides explicit examples.
Firefox’s approach to its in-browser experiments is very much in line with desired and ethical research practices, in so far as we view Mozilla as a privacy-first organization. The availability and “opt-in” to a nightly build is not considered research.
Your contention with not being provided information after the experiment is qualified with “whenever appropriate” and “additional pertinent information”. Debriefing after a deception study is very much appropriate and considered required. However, these are considerations in context of waiving informed consent. I would also point out that “legally authorized representative” in this phrasing refers to people who are legally designated as a representative of the subject, and not the admin of the site in question. For example (broadly), minors cannot legally satisfy “informed consent”, therefore their legally authorized representative, like their parent or legal guardian, are those who sign the forms on behalf of their children. Adolescents may qualify for informed assent. There’s a whole set of additional considerations that experiments much consider for when working with adolescents who reach the age of majority during the research process in context of this.
The waiving of the requirement to document informed consent requires any of the listed qualifications apply, not all. No one is saying that emotional harm is not real, but rather the contention is whether the kind of emotional harm that comes from being forced to log-in to view a website is so significant of a magnitude that it rises above the kind of everyday “harm” experienced in ordinary life. Can you demonstrate that this is the case?
The rest of the discussion about shadowbanning from Facebook and/or other related things are interesting comments but not the point I am making.
Rather, my point is if you replicated the exact same experiment Reddit runs on a University academic setting, it is highly unlikely you will be run out of the university by the IRB from doing so. This was your original claim, arguing that Reddit’s experiment here violates some kind of IRB policy if it were to be run in an academic setting. My point is that, NO, you will NOT be run out of an academic research institute by their IRB for doing something like this, and the way the IRB determines minimum risk and exemptions is part of the reason why, because this specific experiment largely meets many of the metrics for it.
To iterate again, being unable to view the content of a webpage without logging may lead to some discomfort, but this discomfort is not going to rise up to any meaningful level to lead to most IRBs to call it as greater than minimal risk. We can of course twist the situation to make it so, but torturing the very situation to achieve a particular interpretation does not fly with IRB review.
As for Facebook’s psychological manipulation of its users? They can go shove it, honestly.
I don’t like the kind of A/B testing that large corporations do, but I’m not so certain that this particular user experiment Reddit is running would qualify for rigorous IRB review in most academic settings.
Firstly, let’s talk about consent. An IRB can make a determination to waive the requirement to obtain informed consent for research. The IRB must find and document:
(i) The research involves no more than minimal risk to the subjects;
(ii) The research could not practicably be carried out without the requested waiver or alteration;
(iii) If the research involves using identifiable private information or identifiable biospecimens, the research could not practicably be carried out without using such information or biospecimens in an identifiable format;
(iv) The waiver or alteration will not adversely affect the rights and welfare of the subjects; and
(v) Whenever appropriate, the subjects or legally authorized representatives will be provided with additional pertinent information after participation.
Secondly, some kinds of research can be waived from documenting informed consent:
(i) That the only record linking the subject and the research would be the informed consent form and the principal risk would be potential harm resulting from a breach of confidentiality. Each subject (or legally authorized representative) will be asked whether the subject wants documentation linking the subject with the research, and the subject’s wishes will govern;
(ii) That the research presents no more than minimal risk of harm to subjects and involves no procedures for which written consent is normally required outside of the research context; or
(iii) If the subjects or legally authorized representatives are members of a distinct cultural group or community in which signing forms is not the norm, that the research presents no more than minimal risk of harm to subjects and provided there is an appropriate alternative mechanism for documenting that informed consent was obtained.
Insofar as the kind of UI/UX A/B testing which is employed poses minimal risk to the participant and the waiving of the need to obtain informed consent has no adverse effect on the participant, an IRB is likely to make a determination that consent can be waived. It would not surprise me that Universities themselves utilize UI/UX A/B testing for their own websites, both external and internal facing, for improvement. I doubt many would explicitly file a with their IRB to conduct such an experiment, but some may reach out to inquire if there are explicit concerns.
However, at a level even above informed consent, is a question of whether the research is actually subjected to IRB review to begin with. There is a classification of research that is exempt from human subjects review, and some kinds of research do qualify for human subjects review exemptions.
For UI/UX A/B testing, this particular section will have some application, considering a lot of UI/UX A/B testing only cares for aggregate responses to human behavior:
(2) Research that only includes interactions involving educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures, or observation of public behavior (including visual or auditory recording) if at least one of the following criteria is met:
(i) The information obtained is recorded by the investigator in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects;
(ii) Any disclosure of the human subjects’ responses outside the research would not reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects’ financial standing, employability, educational advancement, or reputation; or
(iii) The information obtained is recorded by the investigator in such a manner that the identity of the human subjects can readily be ascertained, directly or through identifiers linked to the subjects, and an IRB conducts a limited IRB review to make the determination required by §46.111(a)(7).
Now, one can make the argument that the kind of profile information websites like Reddit have on you is identifying. In practice, “personally identifying information” has specific definitions and the information that Reddit has on you is unlikely to satisfy that criteria.
Finally, this particular set of charts is a helpful reference for whether research even qualifies for IRB/human subjects review to begin with, and walks through the decision points. Notice that one specifically starts with a question of whether the research contributes to generalizable knowledge…
In short, if you ran the exact same experiment Reddit is running in an University setting for an university website (for example, testing whether making visitors create an university website account [???] to view an article published by the University press versus not having to do so), I doubt you would be run out of the University by the IRB. Perhaps the IRB might have a stern word if you failed to check-in with them prior, but even so I’m not confident that will be the case.
Now with ALL of that said, I still dislike the fact businesses run these experiments. It’s definitely not ethical in the sense that businesses should not be aggressively using its everyday users as guinea pigs for their experiments, but just merely being a shitty thing to do is not sufficient by itself to merit the full IRB process.
The “numbers” are called Discriminators and served a variety of purposes:
You argue that being able to use effectively the same username everywhere is a good thing. The unfortunate reality is the rollout Discord used alongside the limited number of permutations (combinations?) of short usernames makes this impractical. For example, a friend largely goes by a 4-char username, and the switch by Discord means they can’t use that 4-char username on Discord anymore. It’s easy to say like “well, just add something to the end”, but that is exactly what discriminators did.
At the end of the day the benefits weren’t as compelling as the losses (it would suck to have one’s identity impersonated or username stolen, or now most folks with short usernames have to stop friend requests cause they are getting spammed with them, or the fact these accounts are valuable and can be sold).
It is understandable that some people don’t really care about the matter and that’s fine, but it doesn’t make the frustrations other few less important.