What is the story with the study on the electrical shocks?
When a journalist called me a few years ago with an accusation about running a study in 2004 on electrical shocks without approval, I only had a general memory of what happened and as a consequence provided the best answer I could. However, this research took place more than 15 years earlier, and because memory is the way it is, there were lots of things I did not remember at the time.
Thankfully, since then, I had time to go back, collect evidence, and I now know more about this particular case. So, here is my most accurate account of what happened. (Here is also a video, in which I talk with Walter Bender, who was my boss at the Media Lab at MIT and also on the Human Subjects Committee when this thing happened https://www.youtube.com/watch?v=UCsRrg3vj2o).
So, what do I know? I certainly had approval to run this experiment. I had approval for the protocol, for the equipment, for everything. I did approach the MIT ethics board and asked them to find the original protocol, but they said that the only thing that they have now is a screenshot showing a list of all my protocols, including that protocol. protocol 0406000797, Pain and Money. And as can be seen the approval is from June 17th 2004.
I also know that one participant complained about the experiment and that she felt the pain from the electrical shocks was too high. I also know that this complaint initiated an investigation by the committee, and the investigation started they found out that something else was wrong. What was wrong? One of the research assistants running the study did not go through the hour long online training for how to deal with human subjects. Now, this was early in my days at MIT, and I was not fully aware of all the procedures. I assumed at the time that all graduate students passed the training. It turns out they did not. Having someone run a study without doing the hour long online training certainly a violation and I am guilty of that.
The MIT committee decided to prohibit me from supervising more experiments for a year.
So, was it an experiment with electrical shocks? Yes.
Did I have approval? Yes. I had approval for the procedures, for the protocol, for the equipment? Yes.
Did somebody who was running the study not go through the hour-long online training of the subject committee? Absolutely, and sadly, yes. Did I train her on how to run the study? Yes. Did the committee give me an overly due punishment? In my mind, yes. Is this where the story ends? Absolutely yes. Are there people and journalists who are trying to make more of it and make all kinds of accusations? Also, and also sadly, yes.
Pain and Money
Statistical Typo Clarified
In early 2000, James Heyman and I worked on a project related to social norms and market norms. The experiment utilized a non-standard design, and in the analysis, we used an approach called simple cell comparisons. We did this analysis with the software available at the time, called SuperANOVA, in which we compared the means of two cells directly but used the error term from the whole design. We published the paper in 2004 (link here).
One of the peculiarities of this type of design is that the statistics can be reported in terms of an f-value or a t-value, and because it’s a one-degree of freedom comparison, those two approaches are equivalent. The specific numbers of the two approaches would be different but the tests are equivalent.
We published the paper in 2004, and in 2021, we were approached by a group in Hong Kong, who tried to replicate our studies. They managed to replicate the results as we had them in our paper, but they noticed something strange about the f-statistics in the table. They informed the journal and me, and together with the journal editor we wrote a cautionary note about that table. Because the time that passed and the changes in technology and storage I did not have the old data, so we wrote a note to say that something about the table is unclear (see link here).
At the time, it looked to me that we reported f-statistics, but in fact, these were t-statistics. But the editor correctly said that this was speculation, and maybe it’s not worthwhile to speculate that, and then let’s just be extra cautious. And I agreed. When it comes to science, it’s always good to be extra cautious. Our note was published, the cautionary note about this data table.
Interestingly, a couple of weeks later, a good statistician looked at the data and came to the same conclusion. He looked at the t-values and the p-values and basically came to the conclusion that it must have been a change from t to F . he noted that if you looked at the p-values, they correspond to the numbers as if they were t-values, not F-values (see link here).
What does that mean? It means that we had a typo. A serious typo, but a typo nevertheless. We should have indicated that we used t-statistic, and then the table would have been just fine. My guess is that the statistical software pointed out that this was either an F or a t, and we used the F instead of t. And for that, of course, I’m very sorry. But the important thing is that the data holds, the results are statistically significant, and everything is correct in terms of interpreting the results, only that the statistic is a t-statistic and not a F-statistic.
This was a sad and innocent mistake, and I am sorry for it. But, please also think about this: If we looked at all of your work with this level of detail, what % of the time do you think that we would find such innocent mistakes?
Effort for Payment
Letter to the Editors of Psychological Science: Resolving Inconsistencies With Data Gleaning: Regarding Bauer and Ariely (2021)
Expression of Concern: Effort for Payment: A Tale of Two Markets