{"id":1433,"date":"2014-07-28T17:02:30","date_gmt":"2014-07-29T00:02:30","guid":{"rendered":"http:\/\/www.artsjournal.com\/worth\/?p=1433"},"modified":"2014-07-28T17:02:30","modified_gmt":"2014-07-29T00:02:30","slug":"performance-measures-indices-and-rankings","status":"publish","type":"post","link":"https:\/\/www.artsjournal.com\/worth\/2014\/07\/performance-measures-indices-and-rankings\/","title":{"rendered":"Performance measures, indices and rankings"},"content":{"rendered":"

\"Let's<\/a>Ok, not a blog post title likely to set your pulse racing, but with so much discussion in the arts world the past few years on the uses of data, a caveat. It is brought to mind by a story from Money<\/em> magazine (a branch of Time<\/em>), which has tried to make inroads on US News<\/em> turf, and do some college rankings. These new rankings are all about ‘value for money’ – what does it cost to attend; what is the payoff in salary. Whatever your thoughts on that focus, my issue is with how the rankings are constructed.<\/p>\n

Money’s method is given here<\/a>.<\/p>\n

The Chronicle of Higher Education<\/em> assesses the new rankings as follows<\/a>:<\/p>\n

In a world full of frivolous rankings (colleges with the best weather!),\u00a0Money<\/em>\u00a0set out to compile a highly objective one. The result is relatively heavy on outcomes data and light on subjective prestigery like the reputation surveys used by\u00a0U.S. News & World Report.<\/em>\u00a0To develop the rankings,\u00a0Money<\/em>\u00a0joined with\u00a0Mark S. Schneider, a vice president at the American Institutes for Research and a former commissioner of the Department of Education\u2019s National Center for Education Statistics.<\/p>\n

The list\u00a0ranks 665 colleges according to 17 factors in three categories. \u201cQuality of education\u201d includes each college\u2019s six-year graduation rate, student-to-faculty ratio, and a \u201cvalue-added\u201d graduation rate, which reflects the difference between the actual rate and the expected rate based on students\u2019 academic and economic backgrounds. \u201cAffordability\u201d includes borrowing by students and parents, student-loan default rates, and estimates of the average net price of a degree (based on a college\u2019s sticker price, total institutional aid, tuition inflation, and average time to graduation). And \u201cOutcomes\u201d includes various measures of early- and mid-career earnings, based on raw data from Payscale.com.<\/p>\n

Naturally, critics of rankings will find plenty to quibble with. The earnings data, as\u00a0Money<\/em>acknowledges, are self-reported by only those alumni who chose to complete the survey, so it\u2019s not a true measure of an entire class\u2019s average salary. The net-price measure is based on averages, which means the figure might be much higher or lower than what a given student ends up paying.\u00a0The\u00a0\u201dquality of education\u201d measure is based in part on the standardized-test scores of incoming students, a variable that says more about the socioeconomic characteristics of a college\u2019s students than about anything else.<\/p>\n

In short, all rankings are flawed, no matter how much precision they might imply. As one smart person has\u00a0written,<\/a>\u00a0rankings ultimately reflect choices made by real, live human beings: \u201cWho comes out on top, in any ranking system, is really about who is doing the ranking.\u201d<\/p><\/blockquote>\n

Here is the problem. The Chronicle<\/em> focuses on how different outcomes are measured – are they accurate? are they the best measure of the outcome in which you are interested? But that line of criticism suggests that if Money<\/em> had found very, very accurate ways of measuring outcomes, the rankings would be useful. But they would not be. The problem is that the long list of outcomes is turned into an index through a weighting scheme that is entirely arbitrary<\/em>. Accurate data does not solve the problem of how apples are to be added together with oranges. And frozen waffles and bottles of sesame oil. Look at the percentage weights<\/a> given to factors by Money. They are completely drawn from thin air. And so the index is not objective in the least.<\/p>\n

Even if you have a terrific set of accurate data, data that could be very useful to your organization in seeing where you are achieving successes and where you are falling behind, if you assign arbitrary weights in order to come up with an ‘index’, you lose all that useful information in order to come up with one single catch-all result. You will have made the reporting and understanding of the data worse<\/em>.<\/p>\n

Are there implications in the arts? Yes, I’m looking at you, Americans for the Arts<\/a>. It is great that AFTA has collected so much data. But the ‘National Arts Index’ gives us nothing<\/em> of value: much better to look at the various series without trying to aggregate them through an arbitrary scheme (and giving 78 different data sources an equal share in the NAI is completely<\/em> arbitrary).<\/p>\n

Are all indices bad? No, not if there is a rationale for the weighting system. The Consumer Price Index covers prices of all the various items bought by consumers, but the weighting system makes sense: weights are assigned according to the proportion of the average consumer spending budget accounted for by each item. It actually does give us something interesting beyond just providing a list of the thousands of prices we face and how they changed from one year to the next. But the Money<\/em> ranking of colleges, and the AFTA National Arts Index, don’t do that.<\/p>\n

As more and more data come available, it becomes ever more clear: we need to know how to interpret it, and what questions to ask of it. And, as we see here, the limitations on how we can aggregate it.<\/p>\n","protected":false},"excerpt":{"rendered":"

Ok, not a blog post title likely to set your pulse racing, but with so much discussion in the arts world the past few years on the uses of data, a caveat. It is brought to mind by a story from Money magazine (a branch of Time), which has tried to make inroads on US […]<\/p>\n","protected":false},"author":3,"featured_media":1233,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[26],"tags":[],"class_list":{"0":"post-1433","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-issues","8":"entry"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/www.artsjournal.com\/worth\/wp-content\/uploads\/2014\/04\/whirlwind-computer.jpg","jetpack_shortlink":"https:\/\/wp.me\/p3dIW5-n7","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2223,"url":"https:\/\/www.artsjournal.com\/worth\/2017\/07\/the-problem-with-ranking-cities-cultural-vibrancy\/","url_meta":{"origin":1433,"position":0},"title":"The problem with ranking cities’ cultural vibrancy","author":"Michael Rushton","date":"July 10, 2017","format":false,"excerpt":"Two recent publications derive indices to rank different cities according to their cultural vibrancy - from the National Center for Arts Research in the United States, and the European Commission for European cities. They have the same fundamental problem. In each report, a selection of data series applying to cities'\u2026","rel":"","context":"In "issues"","block_context":{"text":"issues","link":"https:\/\/www.artsjournal.com\/worth\/category\/issues\/"},"img":{"alt_text":"not this again","src":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2017\/07\/whirlwind-computer.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1488,"url":"https:\/\/www.artsjournal.com\/worth\/2014\/09\/how-should-we-rank-the-employment-prospects-of-cities\/","url_meta":{"origin":1433,"position":1},"title":"How should we rank the employment prospects of cities?","author":"Michael Rushton","date":"September 1, 2014","format":false,"excerpt":"News sites on the web demand a stream of content, and a sure way to produce something is to make a list. My previous post knocked a recent list of \u00a0'top cities for culture', on the grounds that the way the index was constructed made no sense. Another day, another\u2026","rel":"","context":"In "issues"","block_context":{"text":"issues","link":"https:\/\/www.artsjournal.com\/worth\/category\/issues\/"},"img":{"alt_text":"what are the data telling us?","src":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2014\/04\/whirlwind-computer.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1483,"url":"https:\/\/www.artsjournal.com\/worth\/2014\/08\/how-should-we-rank-the-culturalcreative-scenes-of-cities\/","url_meta":{"origin":1433,"position":2},"title":"How should we rank the cultural\/creative scenes of cities?","author":"Michael Rushton","date":"August 28, 2014","format":false,"excerpt":"Yesterday I came across a ranking of the 'top 20 US cities for culture', from the real estate blog propertyshark.com (no, not one I usually follow, h\/t Ted Gioia). The internet loves to produce listicles, and diminishing returns have long set in when it comes to ranking cities. But this\u2026","rel":"","context":"In "issues"","block_context":{"text":"issues","link":"https:\/\/www.artsjournal.com\/worth\/category\/issues\/"},"img":{"alt_text":"so many creatives!","src":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2014\/08\/NYC-street.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2014\/08\/NYC-street.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2014\/08\/NYC-street.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2014\/08\/NYC-street.jpg?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":2024,"url":"https:\/\/www.artsjournal.com\/worth\/2016\/04\/pricing-for-the-sensible-arts-consumer\/","url_meta":{"origin":1433,"position":3},"title":"Pricing for the sensible arts consumer","author":"Michael Rushton","date":"April 19, 2016","format":false,"excerpt":"At flux[x] Tom Whitwell has a very interesting read on pricing, and the psychology behind how we make choices when confronted with varying sets of prices. I highly recommend it. He uses examples ranging from newspaper paywalls, Netflix and phone packages, wine, and experimental evidence regarding beer. In short: \"Prices\u2026","rel":"","context":"In "issues"","block_context":{"text":"issues","link":"https:\/\/www.artsjournal.com\/worth\/category\/issues\/"},"img":{"alt_text":"ah to be young again","src":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2016\/04\/mateus.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2241,"url":"https:\/\/www.artsjournal.com\/worth\/2017\/11\/soft-power-and-the-arts-13\/","url_meta":{"origin":1433,"position":4},"title":"Soft power and the arts (1\/3)","author":"Michael Rushton","date":"November 10, 2017","format":false,"excerpt":"This week saw the release of two major studies in the UK on culture and soft power: Soft Power Today from the British Council and the University of Edinburgh, and The Art of Soft Power from King's College London. There's a lot of depth to each of the reports, and\u2026","rel":"","context":"In "issues"","block_context":{"text":"issues","link":"https:\/\/www.artsjournal.com\/worth\/category\/issues\/"},"img":{"alt_text":"who would have thought?","src":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2017\/11\/BTS1.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2017\/11\/BTS1.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2017\/11\/BTS1.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2017\/11\/BTS1.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2017\/11\/BTS1.jpg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2017\/11\/BTS1.jpg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":2842,"url":"https:\/\/www.artsjournal.com\/worth\/2023\/03\/really-it-is-ok-for-a-college-to-sell-art\/","url_meta":{"origin":1433,"position":5},"title":"Really, it is OK for a college to sell art","author":"Michael Rushton","date":"March 17, 2023","format":false,"excerpt":"A few days ago I wrote a post re Valparaiso University's plans to sell three paintings, with an estimated total market value of $10 million, to deal with its rather dire financial situation, and to make investments seen necessary to attract students, whose numbers have fallen dramatically over the past\u2026","rel":"","context":"In "issues"","block_context":{"text":"issues","link":"https:\/\/www.artsjournal.com\/worth\/category\/issues\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.artsjournal.com\/worth\/wp-content\/uploads\/2023\/03\/postbox.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/posts\/1433","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/comments?post=1433"}],"version-history":[{"count":0,"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/posts\/1433\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/media\/1233"}],"wp:attachment":[{"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/media?parent=1433"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/categories?post=1433"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.artsjournal.com\/worth\/wp-json\/wp\/v2\/tags?post=1433"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}