This article refers to a real, tongue-in-cheek publication that performs an experiment comparing the efficacy of parachutes to backpacks when jumping off a plane. Contrary to what one would expect, the authors found no significant difference. The reason was because they jumped off a stationary plane.
It’s far too easy for scientists who have already anticipated the outcome of their research to cherry-pick patients and circumstances to achieve the results they expect to see.
We see this all too often. Some scientists are trained to show as strongly as possible the effects of a manipulation. They will go to the bench and optimize the conditions to get a desired effect. If biology deems it to be true, any deviation from those conditions may or may not achieve the same desired effect (such as jumping off a plane at 10000 feet).
Other scientists are trained to test for robustness. They will systematically vary parameters (such as changing the type of plane, the jump altitude, etc.). If time and resources costs deem it possible, scanning those conditions for publication may or may endanger one’s employment.
As quoted in the paper:
I know that people often don’t look detailed enough into what is being investigated to know how to interpret the results of a trial
The problem with science, is that as experiments get more complex and cost-intensive, it becomes a challenge even for experts to properly interpret those experiments. Principal investigators cannot simply rely on their intelligence to interpret experiments that they themselves have never performed.