Most Psychology Papers Fail Replication Test

Discussion in 'Science & Society' started by Yazata, Aug 30, 2015.

  1. Yazata Valued Senior Member

    A team of several hundred international researchers tried to replicate 100 psychological experiments reported in top psychological journals. They were only able to replicate 36% of them. Even when results were successfully replicated, effects tended to be smaller and less dramatic than were originally reported.

    Results varied by psychological specialty. Social psychology papers were only replicated 25% of the time. Cognitive psychology did a little better, with replication in about half the cases.

    Observers are blaming these appalling results on the academic subculture, where an academic's reputation depends on what he or she publishes. So there is a lot of pressure to publish dramatic results that generate talk and rise to the top of the citation rankings. As one of the authors of this replication study writes, "What it takes to be a successful academic is not necessarily that well alligned with what it takes to be a good scientist".
    danshawen likes this.
  2. Google AdSense Guest Advertisement

    to hide all adverts.
  3. sculptor Valued Senior Member

    The worst of the bunch were the "social psychology" "experiments".
    Social psychology is a particularly tricky beasty as there are myriad variables which needs be controlled for.

    Another part of the problem is the publications. publishers don't seem to like phrases like 'most likely' 'high probability' ...etc...etc... .
    So, authors tend to write as though they had definitive answers instead of most likely probabilities.

    When I was studying psyc and doing experiments, we found that tiny changes could seriously impact the findings. Simple things like the color of the room, the arrangement of furniture, lighting, what the experimenter was wearing, male or female, with or without clip board, etc. Trashing other's experiments by changing one or another of the tiny variables became sort of a game. The variables themselves often became subjects of other experiments.
    Caveat: much of what we did circa 1978-79 would not be allowed with today's stricter academic rules. (no one got hurt, but most were grossly misled as to the nature of the experiment for which they had volunteered making "informed consent" meaningless).

    No one study should be viewed as a definitive final outcome/answer.
    Pattern is what we should be looking for, and taking away.
    Don't let non replicability throw you.
  4. Google AdSense Guest Advertisement

    to hide all adverts.
  5. Aqueous Id flat Earth skeptic Valued Senior Member

    It may be that the colleges of psychology have become a bit too removed from the physical sciences to appreciate the value of testing replicability before publishing. And, as the authors suggest, this may be primarily the result of a high value placed on innovation (over replicability, for example).

    The better written remarks (I thought) are in the source abstract. Not surprisingly, the text of the paper is almost exclusively a study in statistics.
    sculptor likes this.
  6. Google AdSense Guest Advertisement

    to hide all adverts.
  7. sculptor Valued Senior Member

    From the study:

    After this intensive effort to reproduce a sample of published psychological findings, how many of the effects have we established are true? Zero. And how many of the effects have we established are false? Zero. Is this a limitation of the project design? No. It is the reality of doing science, even if it is not appreciated in daily practice. Humans desire certainty, and science infrequently provides it. As much as we might wish it to be otherwise, a single study almost never provides definitive resolution for or against an effect and its explanation. The original studies examined here offered tentative evidence; the replications we conducted offered additional, confirmatory evidence. In some cases, the replications increase confidence in the reliability of the original results; in other cases, the replications suggest that more investigation is needed to establish the validity of the original findings. Scientific progress is a cumulative process of uncertainty reduction that can only succeed if science itself remains the greatest skeptic of its explanatory claims.
  8. joepistole Deacon Blues Valued Senior Member

    I remember a similar discussion in a statistics class some 30+ years ago.
  9. sculptor Valued Senior Member

    30 years seems but the blink of an eye.
  10. wellwisher Banned Banned

    Another problem is the scientific method is not designed to deal with things of the unconscious mind. The scientific method was designed to factor these things out.

    As such, the scientific method impacts how the researchers approach their thesis. Most will try to do things in the second and third person and therefore fail to take into account their impact of their own psychology; first person. For example, as mentioned, if they think in terms of the needs of a career, they will unconsciously need to publish, even if it is not fully science. If they seek truth this may seem odd among their peers who are more self centered looking in term of the prestige they can acquire.

    These sciences needs to be updated, with a modified scientific method. For example, if I had a dream this would be considered within the realm of possibility, since the brain generates such data. If I tried to relate the details, this leads us outside of the scientific method, since there is no way to reproduce this data on demand. It can be real an true, but does not follow the rules of the current scientific method. The method is too weak for such data.

    My work around was the scientist needs to become both the observer and the subject, so one can depend on his trained ability to accurately record and interpret unique data when it appears, once. As many scientists gain unique data, then a commonality needs to be found in all the data sources; personality firmware.
  11. wellwisher Banned Banned

    The current approach to psychology is analogous to rational polytheism. Some orientations worship the goddess of sex while others worship a god of power, while others the goddess of feelings. These rational gods are fickle and are not easy to study using the scientific method, but can only be correlated. There needs to be push toward rational monotheism, but it requires a modified scientific method; find what is common to all these gods.
  12. danshawen Valued Senior Member

    This is not a problem. It is an advantage.
  13. Oystein Registered Senior Member

    I believe this to be a certain kind of truth. This life is nothing short of an ennobling uprising of spiritual faith.
  14. danshawen Valued Senior Member

    The replication rate was reported as slightly higher elsewhere (39%).

    Taking into account the many difficulties there are with replication of experimental results in the "soft" sciences like psychology, this is actually higher a replication rate than I expected. Even though these researchers go to great pains to control and prevent random confounding factors creeping into their experiments, and using statistical methodology to filter the outliers that remain, these studies are much more difficult to perform and evaluate without bias than any situation you might encounter in experiments in the hard sciences.

    There is plenty of latitude for criticism of some of the most controversial results obtained in the past. IQ tests are a case in point. For a detailed description of just how spurious the results of psychological experimentation can be, look no further than Stephen Jay Gould's "Mismeasure of Man". Such tests continue to be used in one form or another today, without even an itch of an idea about whether the results are truly valid in a uniform multicultural context or not. What do you think the result would be in replicating just those results, which more or less everyone abuses? Could anyone think of a more damaging misapplication of psychological testing than those would be if they are wrong in their estimation of relative cognitive ability, even in small measure?

Share This Page