In Jonah Lehrer's New Yorker Article (12/13/10 edition), he writes an interesting article on how "well-established, multiply confirmed findings have started to look increasingly uncertain. It's as if our facts were losing their truth..." Drugs losing their effectiveness, wonder if it could be related to IQ testing which suggests that people in 1920s would be considered stupid today (the Flynn Effect)? Frightening thought if the rigors of the scientific method suddenly can't be trusted!
In a 1990 (Johnathon Schooler) study it was found by writing down something it actually decreased ones ability to better remember...it was called verbal overshadowing. Following this study many other areas showed that by putting things into words led to a decrease (so by me writing this blog summary I am effectively going to forget I ever read this...) in the ability not remember things.. Then of course the author of the verbal overshadowing article started relooking at some of his results and as time went by the results to his big ideal started growing weaker.
One of the possible reasons for this decline is simple, "regression to the mean." As more studies are done the sample size increases and thus something that was significant gets averaged out. Another interesting note is that when new exciting research is published articles that tend to buy into this new research are published whereas the null results tend not to get published. This publication bias tends to make people believe the results (dead birds in Arkansas anyone?). Of course bottom line is that often researchers want to be right...their careers and money is often dependent so initial successes are held onto to continue to prove a theory...it is human nature to want to be right (autism and vaccines? amazing how a meme will just hang around). Finally an interesting test was done on trying to replicate an experiment in three different locations. The researcher tried to control every variable and yet the results were vastly different at each site. The implication of this is that often times results and data "are nothing but noise" And what gets published or notice? Yup dramatic results that are statistically significant and unexpected.
In a 1990 (Johnathon Schooler) study it was found by writing down something it actually decreased ones ability to better remember...it was called verbal overshadowing. Following this study many other areas showed that by putting things into words led to a decrease (so by me writing this blog summary I am effectively going to forget I ever read this...) in the ability not remember things.. Then of course the author of the verbal overshadowing article started relooking at some of his results and as time went by the results to his big ideal started growing weaker.
One of the possible reasons for this decline is simple, "regression to the mean." As more studies are done the sample size increases and thus something that was significant gets averaged out. Another interesting note is that when new exciting research is published articles that tend to buy into this new research are published whereas the null results tend not to get published. This publication bias tends to make people believe the results (dead birds in Arkansas anyone?). Of course bottom line is that often researchers want to be right...their careers and money is often dependent so initial successes are held onto to continue to prove a theory...it is human nature to want to be right (autism and vaccines? amazing how a meme will just hang around). Finally an interesting test was done on trying to replicate an experiment in three different locations. The researcher tried to control every variable and yet the results were vastly different at each site. The implication of this is that often times results and data "are nothing but noise" And what gets published or notice? Yup dramatic results that are statistically significant and unexpected.
No comments:
Post a Comment