Friend and colleague (and PhD candidate in Marketing) Jennifer Wiggins adds some details to my rant yesterday about misunderstanding survey results. The article I cited and the sponsoring organization were excited to find that 90 percent of survey participants planned on attending more cultural events in 2004. What they missed was the real finding: that 90 percent of participants wanted to be perceived (and already perceived themselves) as cultured…not that they’d actually follow through. Says Jennifer:
The phenomenon of survey responses where intent and action don’t match up is hugely common in consumer behavior research, and for that matter in any kind of decision-making research. Psychologists and consumer researchers have been trying to figure out for a long time why people don’t do what they say they will do. A few of their findings may shed some light on the situations arts administrators face with this kind of data.First, there is the social desirability element. When responding to self-report surveys, particularly surveys done in person or over the phone, respondents want to give the socially desirable response. Hence, we overestimate how much we exercise and underestimate how much we eat french fries. We also dramatically overestimate how often we go to ‘high culture’ events and underestimate how much time we spend sitting in front of the TV. It’s socially desirable to be ‘cultured,’ and really, who would want to say no to the question, ‘Do you consider yourself to be cultured?’
Second, she says, there’s a common problem of asking broad questions (will you attend more cultural events in 2004?) and making specific conclusions (ballet attendance for the Anytown Ballet Company will rise). We’ll get much more reliable results if we focus our questions as much as possible.
Finally, she wonders if our obsession with finding fresh, new audience members might be blinding us to the more obvious opportunity:
Maybe we shouldn’t be spending so much time and effort trying to attract the people who say they will come but never do. The old adage that it’s much cheaper to keep an existing customer than to attract a new one is very true, particularly when it’s so difficult to figure out who is actually thinking about attending and just needs a little push, and who will really never attend but answers survey questions with the socially desirable response.
The issue here is not whether we should use surveys or not…of course we should. They are essential tools in our learning process as organizations and as managers. But we need to understand their flaws and quirks, like any other tool in the box. If anything, we should use them more, but in quick, informal ways (like having a set of single questions your box office staff asks each customer).
