Measuring only when we mean it

There are all sorts of interesting things about Netflix, and their recent incursion into program development in addition to content delivery. The company that used to mail you DVDs (and killed Blockbuster Video in the process) is now producing and presenting HBO-quality content through the Internet (and making television studios anxious in the process). #

But what caught my attention (at least this month) was their strategy for measuring success in these new ventures. Internally, they’re defining and observing metrics about ”engagement” with their content (which seem to track frequency, duration, and loyalty). Externally, they’re not breathing a word about what they’ve found. #

Reporter after reporter has grilled Netflix Chief Content Officer Ted Sarandos for the numbers. Invariably, he has declined. In fact, he doesn’t even share the metrics with the actors and creative teams who make the shows. #

Why not share the numbers, especially if they claim they’re outstanding? Because there’s no value to Netflix in sharing the numbers out loud. They don’t sell advertising, so ratings don’t draw advertising money. They don’t sell content to cable or media distributors, so ratings aren’t necessary to confirm licensing value. They don’t need numbers to convince viewers to watch. And since they make full-season or multi-season commitments to the creative team, they don’t need to share numbers to signal shows in danger of cancellation. #

Further, they seem to recognize that a new kind of programming model could only be damaged or distracted by an old kind of measurement. #

Here’s why that strategy is particularly interesting to me: #

In the past many months, I’ve been to a LOT of arts conversations, and read a lot online, about measurement and evaluation, particularly around ‘impact’. The ‘how’ of measuring an organization’s impact is fascinating and essential. We all need new tools to explore what we do, and explain how we change the world around us. But the ‘why’ and ‘when’ of impact measurement has often felt squishy. #

As far as I can tell (and I know you will feel comfortable correcting me), you only measure something because that measurement matters to you, because it makes a difference to know it. Further, measurement might matter for two reasons. One, you and your team care about informed internal decisions — you really care about outcomes, and want to be sure you’re continually investing your time, attention, and resources in the best possible way to get there. That’s why Netflix is designing and monitoring ‘engagement’ related to their new productions. Two, you measure something because those metrics will attract specific external resources — audiences will be encouraged to attend because they know something is well attended, funders will provide and continue support on the expectation of certain impact, governments and citizens will invest in something that demonstrates outcomes they value. #

I say specific external resources, because convincing other people to select and favor you over other options is not a generic game. If a specific funder seeks specific evidence of your impact or success, you have reason to measure exactly that thing. If a partner organization is committed to social or civic or educational or personal impact, then again, you might consider developing impact metrics to prove your worth to that partner. If quotes or testimonials convince potential audiences to attend, then by all means gather quotes and testimonials. #

At the moment, however, so much of the impact measurement conversation I’ve experienced feels like prospecting. We’re aware, generically, that some funding sources might want feedback about our work and its impact or outcomes. We assume that if we increase our impact evaluation, that money will flow toward us. But in the meanwhile, we can diffuse our energies and distract our boards in these generic pursuits to make generic points. #

If you care internally about good decisions, and metrics will help you (and they will, if they’re specific), then measure. If you are specifically aware of external value that will flow your way if you can express your impact in specific ways, then measure. If neither of these is true, then really, don’t bother. Measuring won’t make a measurable difference. #

[An extended conversation with Ted Sarandos and members of the Arrested Development creative team is included below. It's really interesting stuff, if you can apportion the time to watch it.] #

#

  #