Every empirical scientist must be concerned with the representativeness of her/his observations. The elections for U.S. presidency in 1948 provide a striking example for the fundamental importance of this issue. The Chicago Tribune declared Harry Truman's Republican opponent Thomas E. Dewey the winner. A historic photograph shows a beaming Truman holding up the newspaper's front page exclaiming in large font “Dewey Defeats Truman”. By contrast, Truman had won!
The false prognosis of the Tribune's poll provided social scientists with plenty food for critical thought. The pollsters had collected answers from a number of its readers sufficiently large to reduce probabilistic uncertainties to reasonable levels. Where did the analysis fail?
The newspaper had polled mainly its readers many of whom preferred the Republican candidate. The polled sample did not represent the opinion of the average American voter. Ever since, social scientists are concerned with building representative samples, striving for the right mix of people in their data sets.
In the social sciences, epidemiology and genetics, data can be collected from thousands of participants, and establishing representative sample sizes is a solvable procedural problem. By contrast, in fundamental life science research accumulating observations on large numbers of study participants or animals may be impractical and burdened with prohibitive cost. Insights often must be gleaned from a few handfuls.
When sample sizes are limited, representative sampling remains elusive. Adding another observation potentially changes outcome. However, novel findings may be confirmed through replication with different methods in diverse species. A finding that is reproducible in different laboratories under varied circumstances sends a powerful message, particularly when it confirms presence or absence of the examined effect. I vividly remember a research paper published in the highly-esteemed scientific journal Science magazine, in which the noted discoveries mainly hinged upon observations from two monkeys and four flying foxes (Calford and Tweedale, 1990).
Irrespective of sample size and representativeness, we must contemplate that we are subject to sampling bias despite our best effort, because we wish for a positive outcome in support of our hypthesis. Particularly in medicine, patients and physicians alike hope for a cure. Wishful thinking may subtly influence our actions. Patients may be selected for suitability. In the rush to use, drugs and procedures may not be tested with sufficient thoroughness. Administering placebos to gravely-ill patients in double-blind studies may seem unethical. Outcome is predicted based on five year experience, though we may have to wait decades to know for sure whether the therapy truly worked. We crave novelty, though the oldest drugs are best understood. We are impatient, believe in success and are inclined to attribute greater value to positive findings.
Yet, continued experimentation tends to diminish the significance of the first findings, eroding the value of derived conclusions. Jonah Lehrer aptly describes this decline effect in his report with the title "The Truth Wears Off" published online in The New Yorker Dec. 13, 2010. Lehrer's contribution should appeal to anyone interested in the reading of statistical data. In his follow-up with the title "More Thoughts on the Decline Effect" posted Jan. 3, 2011, he discusses intriguing reader responses. The posted comments critical of his view provide invaluable insights. I greatly enjoyed pondering this intricate issue. Perhaps, the decline effect may be best viewed as a correction of our initial enthusiasm about a novel finding provided by deepening experience, further refinement of experimentation, and development of new methods of scrutiny.
References
- Calford MB, Tweedale R (1990) Interhemispheric Transfer of Plasticity in the Cerebral Cortex. Science 249:805-807.
