Can statistics lie?!

8202.w1200.r16-9.8515297(1) Image taken from NPO Wetenschap

Imagine drinking a cup of coffee at your grandparents place and suddenly several police officers standing in front of the door screaming that you are under arrest for several attempts to murder and that you have to open the door. This nightmare happened to a Dutch nurse called Lucia de Berk, also known as Lucia B. At that moment Lucia B. worked as a nurse in a hospital specialized in children in The Hague. To make a long story short, 8 years after her arrest she has been acquitted, the explanation for this was due to an incorrect interpretation of statistics. Do you call this phenomenon lying, lack of knowledge or just a misinterpretation? That is up to you! However, to me it is clear that the misinterpretation of statistics can have far reaching consequences. In the following blog I would like to discuss the case of Lucia B. and make you aware that statistics can also “lie”. To make it clear, statistics cannot lie, only humans factors can make statistics lie. The following statement will have a central place in this blog: Can journalists base their story solely on statistical evidence? 

Statistical errors
Nuzz (2014) discusses several statistical errors in his article. Incorrect usage of statistics takes place in several different areas such as in court (Sally Clark), sports and medicine. A common statistical error is related to p-values. Since when do we use p-values? In 1920’s a statistician called Ronald Fischer introduced the so-called p-value. His intention was not to use it as a final test (as we used it nowadays), but he simply wanted to see if the evidence was significant enough so it is worth for a second look. According to Burton, Gurrin and Campbell (1998) you simply cannot accept or reject a null hypothesis based on a p-value, other factors such as confidence interval should not be forgotten. This is a possible explanation why p-values are often misinterpreted. As a journalists or reader there are several ways to judge the quality of a scientific article. As Michèle Nuijten stated in the guest lecture you can have a look at how often the source is cited by others. In addition, I would like you to think about the following question: does the article I have read provide enough information in order to replicate the same study myself? If this is the case you can assume that the statistics are interpreted well. However, it is always advisable to check it yourself. In the lawsuit of Lucia B. the statistical mistakes were caused due to a lack of knowledge of the rules of p-values.

Lies-Damn-Lies-and-then-theres-Statistics-Mark-Twain-620x385
(2) Image taken from Kinder Morgan

Case: Lucia B.
In December 2001 Lucia B. was arrested due to suspicion of murder. Lucia had to work very often when patients ‘spontaneously’ passed out. The question is whether this was a coincidence or not. Several researchers state that the chance is 1 out of 342 million that it could be a coincidence. In other words, according to them there was something else going on. Lucia B. was sentenced for life in jail for seven murders and three attempted. Some stories state that she gave the patients medical overdose. However, till today no evidence can be found that shows that Lucia B. is guilty. Lucia B. has denied everything for years and years. The only ‘evidence’ the court had was statistical evidence. To make a story ‘short’, 8 years later Lucia B. was exonerated from prison due to lack of evidence and incorrect statistical evidence. A few years later Lucia B. received a so-called ‘compensation’ of 45.000 euros and she lived happily every after….

LuciaDeB
(3) Image taken from Filmdoek

What went wrong?
In this case of Lucia B. several researchers replicated the study, made some calculations and came to the conclusion that the results differed from previous research. A possible clarification for this huge mistake, that drastically changed Lucia’s life, is that the statistics were not conducted professionally. The wrong data was used and incorrectly misinterpreted. To be more specific, a multiplication was made of several p-values and this multiplication is mathematically not permitted (for more information see Fisher’s exact test).

Take-home message
From the case above I would like to make you aware that human errors are possible in statistics. Therefore, people should not solely rely on statistical evidence. In my opinion other evidence is necessary before you can draw any conclusion. As shown, statistical results do not always present the truth! According to Marc Seijlhouwer (2011) a trend is becoming more visible:

“More evidence has a base in statistics, rather than any other evidence or the opinion of experts. It is very important that statistics are used and interpreted correctly.”

I believe that it is advisable for journalists to review articles and check if the statistics are correct. Nowadays there exist several tools in order to check if statistical results are correct. Tools like these can for instance calculate if the reported p-values are correct. However, as a journalist you are always depending on what researchers report (researchers can also ‘misinterpret’ data on purpose). Another method to create more transparency and controllability regarding statistical data is to establish an open-source database in which researchers are required to outline their planned investigations and document all their results (Lehrer, 2010). By doing this you might avoid that researchers only report desirable results or other gray practices (Decoster, Sparks, Sparks, Sparks & Sparks, 2015). As Michèle Nuijten explained in the guest lecture, gray practices are statistical and methodological choices that shift a result towards the desired outcome. I would like to end my story with the following quote from Regina Nuzzo (2014):

“P-values, the ‘gold standard’ of statistical validity are not as reliable as many scientists assume.”

 

 

Advertenties

5 gedachten over “Can statistics lie?!

  1. That is definitly very careless, that a judgement is based on wrong calculations and I think that 45,000 Euros are far from being enough compensation for falsely accusing somebody of murder and imprisoning her for years.
    The question is then probably not, if statistics can lie, but if people are unable to calculate and interpret statistical measures. For journalists, who have to write articles about a topic, just using the material scientist provided it is often quite hard to detect mistakes. In this situation scientists just have the control of what information about the study is published and what information they hold back.
    What journalists can do, is consulting experts who may know something about the studies topic and can advise you what may have gone wrong or what information is missing, so that journalists can request more specific information from researchers.

    Liked by 1 persoon

  2. I like the example you chose to explain the subject. The readings of this week made me think much more about human errors in statistics. It’s so weird that so many mistakes are made (i.e. calculation mistakes) and this could happen to you. I agree with Justabark, 45,000 euro’s really not enough to compensate. In addition, in your story, you told that Lucia lives happily ever after. Is that true? Can she find a new job? And did she lost many friends or family because of her prisoning? Furthermore,Your blogpost is easy to read, like the pictures and quotes you used and the structure is well set-up.

    Liked by 1 persoon

  3. Thanks for all your reactions! If Lucia can lives happily ever after, well in my opinion the answer to this question is: NO. The only thing nowadays is that she got ‘famous’ and has her own movie.

    Like

  4. Dit is op Voice of the Vizier herblogden reageerde:
    Nice job. Your write-up once again brings to the fore the dire consequences of misinterpreted statistics. Quite a lot of decision we make daily in real life situations are dependent on test-statistics. Thus, the need and importance of expert opinion cannot be overemphasized in analyzing results from studies. Not to mention standardized procedures that removes all forms of biases and prejudices that could affect the outcome of research endeavours.

    Like

  5. Big compliments to your choice of topic, I was immediately drawn to the story. I wonder why they let the statistics play such a big role in this case. I found a bit more information about this case on the internet and found that they misused some conflicts in the family to make Lucia look more guilty. Apparently, their house burned down when Lucia was still young and even though she wasn’t at home at the time, this was used as ‘evidence’ for her profile as a serial killer. I think that they used the statistics as evidence to get her in jail, as concrete evidence was lacking.

    Like

Geef een reactie

Vul je gegevens in of klik op een icoon om in te loggen.

WordPress.com logo

Je reageert onder je WordPress.com account. Log uit / Bijwerken )

Twitter-afbeelding

Je reageert onder je Twitter account. Log uit / Bijwerken )

Facebook foto

Je reageert onder je Facebook account. Log uit / Bijwerken )

Google+ photo

Je reageert onder je Google+ account. Log uit / Bijwerken )

Verbinden met %s