There was something wrong with psychology. A cascade of warning signs arrived all at once in 2011.

1. 
NEWS 
TECH 
HEALTH 
PLANET EARTH 
STRANGE NEWS 
ANIMALS 
HISTORY 
CULTURE 
SPACE.COM 
  
Live Science  Culture
Psychologists Have a Plan to Fix the Broken Science of
Psychology
By Rafi Letzter, Staff Writer | March 20, 2018 02:40pm ET
 0  0   MORE 


2. Credit: Shutterstock
There was something wrong with psychology. A cascade of warning signs arrived all at once in 2011. Famous psychological
experiments failed, over and over, when researchers re-did them in their own labs. Even worse, the standard methods researchers
used in their labs turned out under close scrutiny to be wishy-washy enough to prove just about anything. Nonsense, ridiculous
claims turned up in major journals. It was a moment of crisis.
The rst sign that it was time for a reckoning, researchers told Live Science, was a single paper published by Cornell psychologist
Daryl Bem in 2011 in the prestigious Journal of Personality and Social Psychology.
The paper discussed nine studies that Bem had conducted over the course of 10 years, eight of which appeared to show powerful
evidence that human beings can perceive things they cannot see or things that have not happened yet. [What Is a Scienti c
Hypothesis?]
His paper presented what looked like real evidence for precognition, "for basically ESP," or extrasensory perception, Sanjay
Srivastava, a research psychologist at the University of Oregon, told Live Science.
For scientists who had dedicated their lives to this science and these methods, it was as if the rug had suddenly been ripped out
from under them.
"With about 100 subjects in each experiment, his sample sizes were large," Slate's Daniel Engber, who has covered the crisis in
psychology at length, wrote in 2017. "He'd used only the most conventional statistical analyses. He'd double- and triple-checked to
make sure there were no glitches in the randomization of his stimuli. Even with all that extra care, Bem would not have dared to
send in such a controversial nding had he not been able to replicate the results in his lab, and replicate them again, and then
replicate them ve more times. His nished paper lists nine separate ministudies of ESP. Eight of those returned the same e ect."
Bem was not a fringe scientist. These were solid results, compellingly demonstrated.
"The paper appeared to be following all the rules of science, and by doing so showed something that almost everybody thought
was impossible," Srivastava said. "And so when that happens you say: Okay, either the impossible really isn't impossible, like maybe
ESP exists, or there's something about how we're doing science that makes it possible to prove impossible results."
In other words, this was, by all the standards available to psychology, good science.
"Unacceptably easy"
Within months of Bem's ESP paper getting published, a trio of researchers at the University of Pennsylvania and the University of
California, Berkeley published a paper in the journal Psychological Science that was in some respects even more disturbing,
according to Simine Vazire, a psychologist at the University of California, Davis.
Joseph Simmons, Leif Nelson and Uri Simonsohn's "False-Positive Psychology" paper demonstrated that, as they put it, "it is
unacceptably easy to publish 'statistically signi cant' evidence consistent with any hypothesis."
It seemed likely that many researchers working with methods they had every reason to believe in had reported results that simply
weren't true. To prove it, they used existing methods in psychology to demonstrate, among other things, that listening to the
Beatles song "When I'm Sixty-Four" makes people a year and a half younger. If psychology worked properly, researchers would
have to accept the proposition that Paul McCartney lyrics have the power to literally shift your birth date.
"A signi cant thing"
Psychology isn't a science of sure things. Humans are weird, and messy, and do things for all kinds of reasons. So, when
psychologists run an experiment, there's always a risk that an e ect they see — whether it's ESP or, say, a tendency to get hungry
when smelling hamburgers — isn't real, and is just the result of random chance. [25 Weird Things Humans Do Every Day, and Why]
But statistics o ers a tool for measuring that risk: the P-value.
"P-value, put simply, is: If everything was just noise, if all the data were random, what are the chances I would have observed a
pattern like the one I observed?" Vazire told Live Science. "What are the chances I would have seen a di erence this big or bigger if
it was just random data?"
If a study has a P-value of 0.01, that means that if there was no real e ect, there would still be a 1 percent chance of getting a result
this big or bigger — a false positive. A value of 0.20 means that even with no real e ect there's still a 20-percent chance of a result
at least this big.
"As a eld, we've decided that if a p-value is less than 5 percent, we're going to treat it as a statistically signi cant thing," Vazire said.
[What Is a Theory?]
If the P-value suggests that a result would only have a 5 percent chance of appearing without a real e ect, it's signi cant enough to
be worth taking seriously. That was the rule in psychology. And it seemed to work — until it didn't.
Decisions
So, with that test in place, how was it "unacceptably easy" to come to false conclusions?




3. The problem, Simmons, Nelson and Simonsohn concluded, was that researchers had too many "degrees of freedom" in performing
their studies. As psychologists conduct experiments, the team wrote, they make decision after decision that can bias their results in
ways P-values alone can't detect.
The P-value test, Vazire said, "works as long as you only compute one P-value per study."
But that's not always how scientists worked.
"If I get a dataset with a dozen or more variables" — things like age, gender, education level or di erent ways of measuring results
— "I can play around with it," Vazire said. "I can try di erent things and look at di erent subgroups."
Perhaps not everyone in a study group reports getting hungry when they smell hamburgers (as in the case of the imagined study
from earlier). But a lot of men ages 30 to 55 do. Scientists might be able to accurately report an apparently statistically signi cant
claim that men in that age range get hungry when they smell hamburgers, and just not mention that the e ect didn't turn up in
anyone else studied.
"If we're allowed to try many times, we're eventually going to get a result that looks extreme, but it's actually by chance," Vazire
said.
And presenting this kind of cherry-picked result just wasn't considered cheating.
"It used to be common practice to look at the data collected during a study and then make decisions," Srivastava said. "Like which
variable is the key test of your hypothesis, or deciding how many subjects to collect."
One way to produce a positive result out of random noise, Srivastava said, is to add subjects to a study in small batches — collect
some results and see if the data o ers the answers you're looking for. If not, add a bit more. Rinse and repeat until a statistically
signi cant e ect emerges, and never mention in the nal paper how many nudges and checks it took to produce that result.
In these instances, most psychologists likely weren't trying to nd false positives. But they are human beings who wanted positive
results, and too often, they made decisions that got them there.
What was planned, and what wasn't?
Once it became clear that the normal ways of doing psychology weren't working, the question was what to do about it.
"I talked a lot about sample size in the beginning, and how we need larger samples," Vazire said.
It's a lot more di cult to fudge the results, whether intentionally or unintentionally, in an experiment performed on 2,000 people
than in a study of 20 people, for example. [What Is a Scienti c Law?]
"That was kind of the rst big push in psychology among people pushing for reform, but eventually it shifted more to
transparency," she said.
And there's where the real pushback began.
"I would say there's pretty good consensus in psychology that we should make our data publicly available whenever possible, and
that we should make our materials and procedures and code — [necessary] to replicate our studies — publicly available."
But increasingly, reformist psychologists — including both Srivastava and Vazire — started pushing for another solution, borrowed
from clinical trials in the pharmaceutical industry: preregistration.
"Preregistration I see as another branch of transparency to let others verify what was planned and what wasn't," Vazire said.
It's a forcing mechanism designed to limit those degrees of freedom Simmons, Nelson and Simonsohn worried about.
"Preregistration means that before you collect data for a study, you write down a plan of what you're going to do," Srivastava said.
"You identify all the things you might have to make decisions about along the way, and you make these decisions in advance." [10
Things You Didn't Know About You]
These decisions include things like what variables psychologists will analyze, how many subjects they'll include, how they'll exclude
bad subjects — all that gets written down in advance and published somewhere with a time stamp so that other researchers can go
back and check it.
The idea is that, without too many degrees of freedom, researchers won't nd themselves drifting toward false-positive results.
"Science in chains"
But not everyone loves the idea.
"There's de nitely a generational di erence," Srivastava said. "When I talk to younger graduate students and early-career people, it
often seems like it just makes sense to them."
That's a highly visible, activist group — preregistration is a hot topic in the online psychology community — and due in part to that
activism, the practice has made signi cant inroads. (The prominent journal Psychological Science now encourages preregistration,
for example.) But preregistration advocates aren't the clear center of power in psychology, and their e orts have encountered
some signi cant pushback.


4. Often, that pushback is uno cial. The controversy appears a lot more heated on Twitter and around psych-department
department water coolers than in the pages of journals. Not too many researchers have publicly staked out anti-preregistration
positions.
But preregistration isn't without its prominent opponents. Sophie Scott, a neuroscientist at University College London and an
expert in the mental processes of speech, wrote a column for Times Higher Education in 2013 titled "Pre-registration would put
science in chains," arguing that the practice "must be resisted."
"Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective," she wrote,
adding that "the requirement to re ne studies and their interpretation prior to data collection would prevent us from learning from
our mistakes along the way."
Scott also argued that preregistration gives too much credit to a narrow kind of scienti c work: hypothesis testing. Not all
scientists work by guring out in advance what questions they want to answer, she wrote, so preregistration would kill exploratory
research.
Vazire acknowledged the concern that preregistration would limit researchers' ability to detect unexpected e ects.
But, she said, "Many of us who push for preregistration say that's not true. You can. All you want. You just have to be honest about
the fact that you're exploring and this was not planned."
Exploratory research, she said, can still be "super exciting and worth publishing," but researchers should be less con dent in its
results."The part of that criticism that is true and I think we need to be really, really clear about is that I will be less con dent in that
result," Vazire said.
"Almost everything I do is exploratory," she said. "I'm just now very upfront about the fact that this is a hypothesis that still needs to
be tested and no conclusions should be drawn yet from it."
"Scientists are human beings"
Advocates of preregistration are quick to acknowledge that it's not a cure-all for the diseases of psychological science.
In 2011, the same year the ESP and false-positives papers came out, Dutch psychologist Diederik Stapel — whose work had shaped
the eld of social psychology — was suspended from Tilburg University for fabricating data in "dozens of studies," according to New
Scientist. It was another signi cant blow, but of a di erent kind than the one for Bem, who seemed to really believe his results
demonstrated ESP.
"Preregistration is not a good check against fraud," Srivastava said. "It's a good check against well-intentioned mistakes and a check
against ordinary human biases and habits."
And, as Vazire pointed out, it's possible to preregister a study incompletely or incorrectly, such that the research still has far too
many degrees of freedom. There are already examples of "preregistered" studies that reformists have criticized for lax and
incomplete registration e orts.
For now, Srivastava said, the project for reformers is to continue to make the argument for preregistration as a route out of
psychology's crisis, and convince their colleagues to follow along.
"One universal is that scientists are human beings," Srivastava said, "and human beings have biases and we have incentives and all
these other things we have to check against."
Originally published on Live Science.

Sanjay Srivastava @hardsci 27 Feb
Replying to @annemscheel
I think we need to move to a standard where the main article
stands on its own. And that means there needs to be enough info
about the preregistration that you don't have to refer to it (it's extra
info but not essential, like supplements and appendices should be)
Sanjay Srivastava
@hardsci
That means instead of just saying "this study was preregistered
[link]" the main article shd say specifically *what* was
preregistered (exclusion rules, scoring, transforms, etc.) and the
results section needs to structurally distinguish between prereg
and non-prereg analyses
10:01 PM - Feb 27, 2018
3 See Sanjay Srivastava's other Tweets
YOU'D ALSO LIKE

5. Advertisement
1 Public Speaking Skills - Dale Carnegie
Training™
Learn The Secrets To Confident & Effective Public Speaking. Download Free Guide!
dalecarnegie.com/Free-eBook
2 Free Big Data Analytics Trends - Get the Free
Whitepaper
The Changes Coming in Big Data Analytics & More. Get the Free Whitepaper Now.
tableau.com
LiveScience 
LiveScience 
LiveScience 
LiveScience 
ESP & Psychic Powers: Claims Inconclusive
Do Scientists Fear the Paranormal?
The New Science of Willpower: Can Self-Control Really Get Used Up?
Men's and Women's Personalities: World's Apart, or Not So Different?
From The Web
The Unusual Link Between Coconut Oil
and Alzheimer's
Memory Repair
You Should Never Shop on Amazon Again
After Seeing This Site
Tophatter
Only Someone with an IQ over 130 Can
Pass This History Quiz
Trendchaser
Ads by Revcontent

6. Advertisement
Science Newsletter: Subscribe
enter email here submit
Follow Us
  
Most Popular
Remember Her? Take A Deep Breath
Before You See What She Looks Like Now
Miss Penny Stocks
This is Why You Should Never Shop on
Amazon Without Using This Trick
Honey
American Picker Mike Wolfe's Mansion Is
Just Plain Disgusting
Miss Penny Stocks
After Death, Your Microbiome Could Still Help the Living
If 'Oumuamua Is an Alien Spacecraft, It's Keeping Quiet So Far
If You Want Your Friend to Vaccinate Their Kids, Don't Try to Change Their Mind
Greenland Has a Mysterious 'Dark Zone' — And It's Getting Even Darker

7. Advertisement
Advertisement
65 Pounds of Plastic Trash Tore This Whale Apart from the Inside

8. Home About Us
enter email here ...
FOLLOW US
SUBSCRIBE
SUBMIT
Copyright © 2018 All Rights Reserved.
     