Imperfect Surveys and Focus Groups

Raymond Chen has an excellent article on the various way that people distort surveys done for product focus groups. People often say what they think the interviewer wants to hear or the perspecitive that they like to think they have rather than the truth. There is another aspect of systematic bias that can occur in this type of situation - bias from the people operating the survey. This is a lot easier to occur (intentionally or otherwise) in the form of biased questions, samples and environment. This is also very prevalant in political polling - it's always struck me as odd that there is such a thing as a republican pollster. In the current presidential campaign, all these factors come into play. In particular, the likely voters that everyone is trying to measure. Problem #1 - if you didn't vote in the last election (say if you couldn't vote because you weren't 18 yet) then your not a likely voter. Problem #2 - only have a cell phone or maybe you don't want to hear from telemarketers and don't pick up the phone at night? Then you won't be called and your not part of the poll. As you might gather, this tends to skew polls to older voters. Of course these people know the polls are intrinsically biased, they just don't care - polls are just another form of PR at this point. If you tell enough people that candidate X will win, then people who are for candidate Y won't bother to vote. The Democratic primaries were a good example of how divorced from reality polls can be.