Secrets of success

a big data setAt Salon, Laura Miller writes about literary scholar Franco Moretti, and his efforts to analyze texts in order to discover what makes a success:

Miller: One of the aspects of your work that’s the most counterintuitive at first glance is that you’re not that interested in studying literary masterpieces. You study literary works in large masses, regardless of whether they’re good or not. You’re not looking at “Middlemarch”; you’re looking at 7,000 mostly mediocre Victorian novels. Why is that interesting to you?

Moretti: First of all, those novels were there, it’s just that there wasn’t the desire to understand what those 7,000 — or, rather, 6,900, if we don’t count the ones that are still being read today — authors had in mind when they were writing. Why do so many people write things that others don’t like to read in the end? What is going on?

It’s a good question! The problem is, I don’t think it can be done. It is not that there are not patterns to be found in the texts of works that have lasted and works that have not. When you take any large data set and look for correlation you are bound to find something of ‘statistical significance’.

I see the problem as being that there is simply so much noise in the data. What succeeds and what does not in the arts often depends on which works, in a process determined with a large degree of chance, capture enough of a small critical mass of support to launch a cascade, where readers (or listeners) follow what the crowd has begun to follow. This isn’t to say that bad works become popular due to fads (although we all have our own examples where we think that has occurred!), but it means that out of many good works only a few will really ‘catch on’, and that there is a large degree of randomness in what those works will be. The search for phrases or structures in texts that became what we now consider classics has a high probability of finding correlations that are spurious, i.e. falsely suggesting that we have discovered the answers.

A very good, not so high-tech survey of research into cascades is:

Bikhchandani, Sushil, David Hirshleifer, and Ivo Welch. 1998. “Learning from the Behavior of Others: Conformity, Fads, and Informational Cascades.” Journal of Economic Perspectives, 12(3): 151-170. (free download here).

You might also look at Duncan Watts’ 2007 article from the New York Times Magazine, “Is Justin Timberlake a Product of Cumulative Advantage?


Leave a Reply

Your email address will not be published. Required fields are marked *