(1) Image taken from NPO Wetenschap
Imagine drinking a cup of coffee at your grandparents place and suddenly several police officers standing in front of the door screaming that you are under arrest for several attempts to murder and that you have to open the door. This nightmare happened to a Dutch nurse called Lucia de Berk, also known as Lucia B. At that moment Lucia B. worked as a nurse in a hospital specialized in children in The Hague. To make a long story short, 8 years after her arrest she has been acquitted, the explanation for this was due to an incorrect interpretation of statistics. Do you call this phenomenon lying, lack of knowledge or just a misinterpretation? That is up to you! However, to me it is clear that the misinterpretation of statistics can have far reaching consequences. In the following blog I would like to discuss the case of Lucia B. and make you aware that statistics can also “lie”. To make it clear, statistics cannot lie, only humans factors can make statistics lie. The following statement will have a central place in this blog: Can journalists base their story solely on statistical evidence?
Nuzz (2014) discusses several statistical errors in his article. Incorrect usage of statistics takes place in several different areas such as in court (Sally Clark), sports and medicine. A common statistical error is related to p-values. Since when do we use p-values? In 1920’s a statistician called Ronald Fischer introduced the so-called p-value. His intention was not to use it as a final test (as we used it nowadays), but he simply wanted to see if the evidence was significant enough so it is worth for a second look. According to Burton, Gurrin and Campbell (1998) you simply cannot accept or reject a null hypothesis based on a p-value, other factors such as confidence interval should not be forgotten. This is a possible explanation why p-values are often misinterpreted. As a journalists or reader there are several ways to judge the quality of a scientific article. As Michèle Nuijten stated in the guest lecture you can have a look at how often the source is cited by others. In addition, I would like you to think about the following question: does the article I have read provide enough information in order to replicate the same study myself? If this is the case you can assume that the statistics are interpreted well. However, it is always advisable to check it yourself. In the lawsuit of Lucia B. the statistical mistakes were caused due to a lack of knowledge of the rules of p-values.
(2) Image taken from Kinder Morgan
Case: Lucia B.
In December 2001 Lucia B. was arrested due to suspicion of murder. Lucia had to work very often when patients ‘spontaneously’ passed out. The question is whether this was a coincidence or not. Several researchers state that the chance is 1 out of 342 million that it could be a coincidence. In other words, according to them there was something else going on. Lucia B. was sentenced for life in jail for seven murders and three attempted. Some stories state that she gave the patients medical overdose. However, till today no evidence can be found that shows that Lucia B. is guilty. Lucia B. has denied everything for years and years. The only ‘evidence’ the court had was statistical evidence. To make a story ‘short’, 8 years later Lucia B. was exonerated from prison due to lack of evidence and incorrect statistical evidence. A few years later Lucia B. received a so-called ‘compensation’ of 45.000 euros and she lived happily every after….
(3) Image taken from Filmdoek
What went wrong?
In this case of Lucia B. several researchers replicated the study, made some calculations and came to the conclusion that the results differed from previous research. A possible clarification for this huge mistake, that drastically changed Lucia’s life, is that the statistics were not conducted professionally. The wrong data was used and incorrectly misinterpreted. To be more specific, a multiplication was made of several p-values and this multiplication is mathematically not permitted (for more information see Fisher’s exact test).
From the case above I would like to make you aware that human errors are possible in statistics. Therefore, people should not solely rely on statistical evidence. In my opinion other evidence is necessary before you can draw any conclusion. As shown, statistical results do not always present the truth! According to Marc Seijlhouwer (2011) a trend is becoming more visible:
“More evidence has a base in statistics, rather than any other evidence or the opinion of experts. It is very important that statistics are used and interpreted correctly.”
I believe that it is advisable for journalists to review articles and check if the statistics are correct. Nowadays there exist several tools in order to check if statistical results are correct. Tools like these can for instance calculate if the reported p-values are correct. However, as a journalist you are always depending on what researchers report (researchers can also ‘misinterpret’ data on purpose). Another method to create more transparency and controllability regarding statistical data is to establish an open-source database in which researchers are required to outline their planned investigations and document all their results (Lehrer, 2010). By doing this you might avoid that researchers only report desirable results or other gray practices (Decoster, Sparks, Sparks, Sparks & Sparks, 2015). As Michèle Nuijten explained in the guest lecture, gray practices are statistical and methodological choices that shift a result towards the desired outcome. I would like to end my story with the following quote from Regina Nuzzo (2014):
“P-values, the ‘gold standard’ of statistical validity are not as reliable as many scientists assume.”