Monday, June 18, 2012

selection bias

Selection bias is selecting a sample that is likely to favor one proposition over another. For example, by asking only those who have bought Chevrolets which car is better, Chevy or Ford, you will most likely get overwhelming support for Chevy. By asking only those who have made a full recovery after back surgery whether back surgery is a good option for people with back problems, you are likely to get overwhelming support for the surgery. If you ask only those whose back problems continued after the surgery, you are likely to get an overwhelming response against the surgery.

Selection bias partly explains why there are reports of many satisfied customers who go to psychics, tarot card readers, palm readers, faith healers, acupuncturists, homeopaths, and others who provide bogus treatments such as mistletoe for cancer. The unsatisfied customers are either not asked for their opinion, they're too embarrassed to give it, or they're dead.

Edzard Ernst, M.D., who was trained in various non-conventional medical therapies, provides an example of selection bias that occurred while he was studying the therapeutic effects of mistletoe injections on cancer patients. He was told that the effect would be a lessening of suffering.

Whenever I gave mistletoe injections, the results seemed encouraging. But young doctors are easily impressed, and I was no exception. What I didn't appreciate then was a relatively simple phenomenon: the hospital where I worked was well known for its approach across Germany; patients went there because they wanted this type of treatment. They were desperate and had very high expectations - and expectations can often move mountains, particularly in relation to subjective experience and symptoms. We call this "selection bias". It can give the impression that a therapy causes a positive health outcome even when it has no positive action of its own.

Patients receiving questionable treatments or clients seeking advice from questionable soothsayers are highly motivated to be helped and to have the healer or reader succeed. Such people are often extremely generous in their efforts to personally validate the words, images, or advice of the reader/healer. Some will even assent to claims they know are false, as one of Gary Schwartz's subjects did with a medium who got the subject to agree that her husband was dead when in fact he was still alive. Schwartz engaged in selection bias again when he omitted much of his data in published papers in the Journal of the Society for Psychical Research that supported the hypothesis of survival of consciousness after death. In his book, The Afterlife Experiments, he describes numerous subjects in his experiments who are conspicuously not mentioned in the published papers on those experiments. Rupert Sheldrake showed selection bias when he omitted 40% of his data in a study claiming to provide statistical evidence for the psychic abilities of a parrot.

The best way to avoid selection bias regarding questionable treatments and various divination techniques is to randomize samples, use control groups, and double-blind experiments. The best way to reduce selection bias by scientists is to expose its occurrence and publicly chastise offenders.

Skeptics and parapsychologists have accused each other of selection bias in determining which studies to include in the ganzfeld meta-analysis. Skeptic Ray Hyman did the first meta-analysis of 42 ganzfeld experiments and found no evidence of ESP. Parapsychologist Charles Honorton, on the other hand, found evidence of "anomalous information transfer." In 1994, Daryl Bem and Honorton published the results of a meta-analysis of 28 ganzfeld studies and once again found evidence for anomalous information transfer. In 1999, Julie Milton and Richard Wiseman published their own meta-analysis of ganzfeld studies and concluded that "the ganzfeld technique does not at present offer a replicable method for producing ESP in the laboratory." Much of the disagreement in analysis centered on what criteria to use in deciding which studies to select for the meta-analysis.

selection bias in polls and surveys

Researchers can bias the results of polls and surveys by using a biased method of selecting subjects for their study. Selecting subjects from a non-representative section of a population is a common way to bias a sample. Using samples that are too small to be representative is a frequent error made by researchers.
Misconceptions of chance are not limited to naive subjects. A study of the statistical intuitions of experienced research psychologists revealed a lingering belief in what may be called the “law of small numbers,” according to which even small samples are highly representative of the populations from which they are drawn. The responses of these investigators reflected the expectation that a valid hypothesis about a population will be represented by a statistically significant result in a sample with little regard for its size. As a consequence, the researchers put too much faith in the results of small samples and grossly overestimated the replicability of such results. In the actual conduct of research, this bias leads to the selection of samples of inadequate size and to overinterpretation of findings. Kahneman, Daniel (2011-10-25). Thinking, Fast and Slow (pp. 422-423). Macmillan. Kindle Edition.
Alfred C. Kinsey’s famous studies on sexual behavior in the 1950s have been repeatedly cited as the basis for the claim that 10% of the population is gay. This statistic has been widely cited in both the mass media and in scientific publications, though it is based on a biased selection of samples. Numerous studies have been done since Kinsey's work was published and these later studies put the percentage of adults who describe themselves as exclusively gay as much lower than the 10% figure. Some have found the rate to be between one and two percent.* It should be noted, however, that "survey research methodologies often result in underreporting of stigmatized behaviors."*

Kinsey gathered his data, in part, by distributing questionnaires to prisoners and to people who attended his lectures on sexuality, neither of which was likely to be a good cross section of Americans (Carroll 2005: 140). For his studies on male sexuality, "he interviewed only white men, and these respondents were disproportionately from lower socioeconomic classes."*

A study that considered attraction to the same sex in measuring homosexuality found "8.7, 7.9, and 8.5% of males and 11.1, 8.6, and 11.7% of females in the United States, the United Kingdom, and France, respectively, report some homosexual attraction but no homosexual behavior since age 15."*

In 1994, sociologist Edward Laumann headed a team of sociologists that studied U.S. sexual behavior. They interviewed a representative sample of the U.S. population between the ages of 18 and 59. Laumann found that over a five-year period, 4.1 percent of U.S. men and 2.2 percent of U.S. women had sex with someone of their own sex. If the time period is extended to include their entire lives, these totals increase to 7.1 percent of the men and 3.8 percent of the women.*

Paul and Kirk Cameron reported in 1998: "The 1994 University of Chicago 'definitive' survey of adults estimated prevalence of homosexuality among males at 2.8% and among females at 1.4%. Corrected for the exclusion of those over the age of 59 years, the estimates should be 2.3% and 1.2%."* A study in Britain in 2000 found that about 2.6% of men and women reported having had a same-sex partner within the previous five years and 8.4% of the men and 9.7% of the women reported having had at least one sexual experience with a member of the same sex.*

One wonders, however, if anything approaching unbiased data is possible for determining what percentage of any human population is homosexual. Given the long history of religious prohibition of homosexuality and the widespread revulsion of homosexual behavior that has often led to torment and persecution, it is likely that researchers in this area will be motivated by something other than a genuine search for the truth. Results will differ depending on how one defines 'gay,' 'lesbian,' and 'homosexual'. Methods of gathering data samples will vary widely and the participants in such studies may not be highly motivated to reveal much about their sex lives.

There is some irony in the fact that the Kinsey studies are cited as the source of the statistic that 10% of the population is gay. As Michael Shermer notes, Kinsey made it clear that he did not believe human males "represent two discrete populations, heterosexual and homosexual." Kinsey maintained that "it is a fundamental of taxonomy that nature rarely deals with discrete categories. Only the human mind invents categories and tries to force facts into separate pigeon-holes" (Shermer 2005: 246). Nature has a bias toward variation. The idea that people should fall into neat categories as 'gay' and 'straight', or even 'male' and 'female', is not consistent with the lessons of evolution. Any study that creates such false dichotomies will be misleading.


Many politically biased websites and organizations poll their readers or members and then pass on the data as if it were representative of the general population. Norman Bradburn, former director of the National Opinion Research Center at the University of Chicago, coined the acronym SLOP to describe polls that use selection bias to get their samples. SLOP is an acronym for self-selected listener opinion polls. Bradburn compares SLOP to radio talk shows: they attract a slice of America that is not representative of the country as a whole. “As a result, SLOP surveys litter misinformation and confusion across serious policy and political debates, virtually wherever and whenever they are used” (Richard Morin, “Call-in polls: Pseudo-Science debases journalism,” Sacramento Bee, Feb. 12, 1992, p. B11).

The inaccuracy of such polls should be obvious. Those who call in to give their opinion are self-selected rather than randomly selected. It appears that people who are willing to call in their opinion will sometimes call in their opinion more than once. For example, in a USA Today call-in poll 81 percent of the more than 6,000 respondents said that “Donald Trump symbolizes what made the U.S.A. a great country.” However, 72 percent of the favorable calls came from two telephones in one insurance company office.

CBS tried the gimmick of call-in polling in “America on the Line,” which featured two surveys conducted immediately after President [George Herbert Walker] Bush’s State of the Union speech. There were 314,786 self-selected callers in one survey and 1,241 adults previously selected by a more scientific method in the other survey. The latter was to act as a check on the call-in survey. CBS’s Dan Rather commented on the similarity of results in the surveys, a sentiment that was echoed the next day in the Washington Post, which wrote, “by and large, the two polls produced the same or similar results.” The facts, however, do not support this judgment. “On two of the nine questions asked in both polls, the results differed by more than 20 percentage points. On another five, the differences were 10 percentage points or more” (Morin, cited in Carroll 2005: 147).

 For more on how proper polling and sampling should be done click here.

2 comments:

  1. Hello everyone i Am williams pater and i am from USA i am here to give my testimony about an herbal doctor called Dr,olu I was heartbroken because i had very small penis,not nice to satisfy a woman, i have been in so many relationship, but cut off because of my situation, i have used so many product which doctors prescribe for me, but could not offer me the help i searched for. i saw some few comments on the internet about this specialist called Dr,OLU and decided to email him on his email i saw on the internet,(drolusolutionhome@gmail.com ) so I decided to give his herbal product a try. i emailed him and he got back to me, he gave me some comforting words with his herbal product for Penis Enlargement, Within three weeks of me use it, i began to feel the enlargement, " and now it just 4 weeks of using his products my penis is about 8 inches longer, and i had to settle thing out with my ex girlfriend , i was surprised when she said that she is satisfied with my performance in bed and i now have a large penis.thanks to DR OLU for is herbal product. you can also reach him with emsil drolusolutionhome@gmail.com though is..number WHATASPP him today on this number [ +2348140654426 ]

    ReplyDelete
  2. My dad was diagnosed with Parkinson's disease his symptoms were shuffling of feet,slurred speech, low volume speech, degradation of hand writing, horrible driving skills, right arm held at 45 degree angle, but now he finally free from the disease with the help of total cure from ULTIMATE LIFE CLINIC, he now walks properly and all symptoms has reversed, he had trouble with balance especially at night, getting into the shower and exiting it is difficult,getting into bed is also another thing he finds impossible.we had to find a better solution for his condition which has really helped him a lot,the biggest helped we had was ultimate life clinic they walked us through the proper steps,am highly recommended this www.ultimatelifeclinic.com to anyone who needs help.

    ReplyDelete