Do you believe in magic?
I (well Leo actually) came across an interesting study published just this year in the rather magnificently titled World Journal of Surgery (kind of puts every other journal of surgery in its place doesn’t it?). The study that looks at whether phases of the moon, zodiac signs and Friday the 13th have any relationship with surgical outcomes over a 9-year period in a hospital in Germany. Considering myself a rational creature and armed with some pieces of paper saying I’m a scientist I felt a bit of smug satisfaction on reading that they found no relationship. But it also got me pondering. The same study reported that more than 40% of health professionals believe that lunar phases are influential, and most of these people are presumably armed with similar pieces of paper that I have.
Of course society-wide such beliefs are probably even more universal, for example; in some cultures national birth-rates are affected by astrological signs. To some extent, who cares, right? If you want your kid born in the ‘year of the dragon’ but not in the ‘year of the horse’ good luck to you, to be honest I can think of worse reasons to have a child at a particular time. But what implication does this have for us as researchers?
It appears people are, if not conditioned, then particularly inclined to place quite a lot of stock in superstitious beliefs, regardless of how tenuous the rationale behind them might be. People also try to find and interpret ‘evidence’ to support what they already think, to reinforce already-held beliefs. That has implications for what people make of the research we publish, but importantly also for the way that researchers go about their own business – given that a large proportion of researchers are people too. If we take this one smallish step further, there is the question of how our personal beliefs and biases influence our own research, whether it is a superstition or religious conviction, or beliefs about a theory, a treatment, or a relationship between variables etc.
It is a truism that statistics can be used to support just about any story you choose, if they are manipulated the right (wrong) way. A paper by Szydlo and colleagues provides a nice example of one way you can show an association between variables when actually there is none. To stay on theme, they looked at the association between zodiac sign and survival rates from a particular type of operation. They found no association when the data were analysed correctly (as per the pre-planned protocol) but they also identified a 37% increased risk of poor outcome for Aries, Taurus, Gemini, Leo, Scorpio, and Capricorn compared to the other signs when they performed some gentle statistical gymnastics. Interestingly (unfortunately) the authors stopped short of explicitly suggesting that other studies that they cite - which support the zodiac garbage - had done something similar but I think I can guess what their view would be. The point is that if you want to find something, you can, you just have to massage the numbers the right way.
Obviously the example here is pretty extreme, given that while it’s probable more than a few of us read our star signs in the newspaper every now and then for shits and giggles, not many are likely to actually research the issue. I think there is a worthwhile message though, a real reminder of how crucial it is to set down very specific research and analysis plans before we get our hands into the data. Registration and publication of protocols for large studies is undoubtedly a good thing and a nice check in the system, but this often doesn’t cover secondary analyses which are usually not as clearly defined and planned. Even if a study protocol is not to be published, that doesn’t pbviate the need for it to be detailed in advance and used to guide conduct of the study. Maybe we don’t believe in zodiac signs, leprechauns, or the all-seeing eye but that doesn’t put us above the influence of pre-existing beliefs and preconceived ideas that can influence our work. Recognising the possibility and taking steps to minimise possible bias is a crucial part of conducting good science.
Jochen Schuld, Jan E. Slotta, Simone Schuld, Otto Kollmar, Martin K. Schilling, Sven Richter. Popular Belief Meets Surgical Reality: Impact of Lunar Phases, Friday the 13th and Zodiac Signs on Emergency Operations and Intraoperative Blood Loss. World Journal of Surgery. 2011; 35:1945–1949
R.M. Szydlo, I. Gabriel, E. Olavarria, and J. Apperley. Sign of the Zodiac as a Predictor of Survival for Recipients of an Allogeneic Stem Cell Transplant for Chronic Myeloid Leukaemia (CML): An Artificial Association. Transplantation Proceedings. 2010; 42;3312–15.