This post makes me just a little sad to write. Chorus America, a while ago, published the results of a study, which they say shows that people who sing in choruses are exceptionally good citizens. They then say that choruses should bring this information to the media, “to help establish an awareness of the personal and communal benefits of choral singing.” Here’s their press release about the study, and here’s the study itself. (The quote comes from the end of the study.)
So why am I sad to talk about this? Because the study suffers from an elementary misuse of statistics. But it’s so eager, so hopeful, and so well-intentioned that it’s almost painful to tell the poor choral people that their work is flawed. I feel like I’m telling a lovely child that I can’t take her to the zoo today.
The claims the study makes are very strong. Here are excerpts from the press release:
If you enjoy singing with your neighbors, congregation, or classmates, you’re taking an increasingly popular path to a successful life….Seventy-eight percent of choral singers indicated they “at least sometimes” volunteer their time in their community, while only 50% of the general public say the same….Choral singers donate 2.5 times more money to philanthropic organizations than the general public….Ninety-six percent of choral singers surveyed who are eligible voters said they vote regularly in national and local elections; only 70% of the general public cites the same level of participation.
Choral singers, in other words, are better citizens.
What’s wrong with this? Well, first, if you find that choral singers participate more in civic activities, you’ve loaded the dice, because simply by singing in a chorus, they’re participating in something. Your sample, in other words, is — by its very nature — a sample of people who already participate in at least one civic activity. It might not be remarkable to find that they participate in other things as well. (If I told you that that people who buy season tickets to the NFL like sports more than their neighbors do, would you be surprised?)
But that’s only the start of the trouble. The biggest problem is the control group (so to speak) for the study, the people whom choral singers are compared to. It’s the general public! Or in other words, the entire population. Choral singers vote more often than the general public does? They give more to charity? They volunteer more? Well, so what? Suppose I told you that choral singers commit fewer murders than a random sample from the population at large. You wouldn’t be surprised. You might say, “But of course that’s true, because we’re pretty sure that, by and large, the kind of people who commit murder don’t sing in choruses.”
So maybe the truth about voting, volunteering, and charitable giving is similar. Maybe the facts cited in the study aren’t really facts about choral singers, but instead facts about the demographic that choral singers come from.
Maybe these aren’t facts about choral singers, but instead facts about the demographic that choral singers come from. Maybe the kind of people who sing in choruses — measured by standard demographic traits, like income, education, or occupation — are already the kind of people who vote, volunteer, and give to charity. Or maybe, to look at this a different way, people who participate in any community activity are more likely to vote and give to charity. Which would mean choral singers weren’t exceptional, because they were behaving exactly the way any people who get out of the house to do anything in their community would behave.
What the study should have done is so elementary that — to sing this song again — it makes me wince to set it forth. The study should first have found out what kind of people sing in choruses, what their other demographic traits might be, and how, as a group, people from that demographic fare in all the measures that we might want to apply to choral groups.
Then theh study should have looked at people who participate in other community activities, and find out their characteristics. Only then should it have looked at choral singers, and found out if they — as a distinct group within their demographic, and as compared to others from that demographic who take part in community activities — have any special traits.
As I said, all this is elementary. The conclusions of the Chorus America study might in fact be true. Maybe choral singers really are exceptional citizens. But Chorus America hasn’t even come close to proving that.
(I ascribe this lack of skills and judgment, by the way, to the same eagerness about the arts that makes arts advocates sometimes think uncritically in other ways. See, for instance, my post about the arts bailout, my Wall Street Journal essay on the same subject, and my post about the none too grounded proposals that came out of last summer’s mammoth arts conference in Denver.)