Rating the Ratings

It’s easy to guess who’s going to end up on top: Harvard, Yale and Princeton round out the first three [spots] essentially every year. In fact, when asked how he knew his system was sound, Mel Elfin, the [US News & World Report] rankings’ founder, often answered that he knew it because those three schools always landed on top.
Nicholas Thompson, “The Best, The Top, The Most,” The New York Times, Education Life Supplement, pg. 24. August 3, 2003.

Long before most teens sit down to compose those onerous admission essays, most parents have been exposed to the peculiarly American practice of ranking our institutions of higher education. Undoubtedly, college ranking is both a distinctly American practice and, from a dispassionate viewpoint, a peculiar one. It fits a national profile that values immediate, pragmatic answers above traditional responses. It also reflects the premium that Americans place on winning at the expense of social solidarity. And, in these characteristics, college ranking is not an isolated activity.

Market Comparisons

College Ranking v. Game Shows

A national news magazine is not the only player in the lucrative arena of academic merit. Brainiac programs also appear to be doing well on television. There are contests between individuals (The Scripps National Spelling Bee) and between institutions (The National College Bowl). High brow viewers tune in to the long-running, syndicated powerhouse, Jeopardy. The far side of the decorum spectrum offers up weekly fodder for thought on Are You Smarter Than a Fifth Grader? Formats differ, but the message is the always same: when it comes to book smarts—and “book smart” is a very American concept—there are clear winners and losers. By the end of the game, any culturally conversant audience member can tell the one from the other. Why should picking a winning college be different?

College Ranking v. The Big Mac

The answer ought to be obvious. Real life is fraught with complexities and uncertainties that defy the rules of the most sophisticated games—or mathematical models. However, in the United States, the appropriate domain for binary outcomes (win/lose events) started to grow fuzzy about twenty-five years ago. To our colonial pragmatism and Industrial Era drive, market factors of the Eighties added a super-sized portion of product branding. The result has been an all-purpose recipe for scoring, counting and grading everything imaginable, or not. Ranking the scientifically unmeasurable is as much a part of the late twentieth century American legacy as the opportunity to buy a Big Mac in Moscow.

College Ranking v. 94 Point Wine

About the time that US News and World Report began publishing its first college lists, an unemployed Maryland lawyer named Robert Parker started ranking bottles of wine on a scale of one to one hundred. By his own judgement, Parker had a sensitive palate and a taste for the grape, although he lacked any credentials to develop reliable metrics. Old world winemakers were appalled by this demonstration of American reductionism, but consumers in the US have embraced Parker as the guru and governor of all things vinicultural.

If you, too, accept that the tingling papillae of a besotted attorney provide a sound basis for assigning a number to a bottle of wine, then you may also support the use of a proprietary (top secret) scale to rank one American university four points higher than another institution of comparable research and teaching productivity. But, this judgement may appear unduly harsh. Actually, there is nothing wrong with scoring wines or colleges, if everyone involved acknowledges two factors. First, highly complex beverages and organizations are not game shows with established rules of play. Second, the methods of Parker and of US World & News Report do not adhere to the cardinal scientific principle of replicability.

Up Close and Personal

Each year US News and World Report publishes multiple lists of one-size-fits-all ratings for American colleges and universities. Educational institutions deemed comparable by the raters (and not necessarily by anyone else) are grouped together into tiers (e.g., first, second, third) within a particular category—National Research Universities, Regional Masters’ Universities, National Liberal Arts Colleges, Regional Comprehensive Colleges. Once all institutions have been grouped into categories and tiers, seven aggregate factors provide the basis for rating each individual institution in relation to its predetermined peers. These seven factors are: peer assessment, student retention, faculty resources, applicant selectivity, financial resources, graduation rate, and alumni giving.

Using a mathematical formula that is not subject to public or governmental review, the researchers weight their seven factors. The result is that some factors will have a greater impact than others in the overall ranking of an institution. Despite the proprietary nature of the computational formula, two aspects of the weighting schema are (technically) public knowledge—which is not the same as making them widely understood. Anyone relying upon these rankings during a college search should be aware of these two aspects of weighting and their implications.

The first aspect of concern is the annual variability of the weighting formula itself. This means that the ratings are not constructed according to an unchanging mathematical formula, but are built anew each year. Because the formula is not fixed, valid comparisons cannot be made over time. When faced with a school that jumps four or five places on a list from one year to the next, the end user has no basis to conclude that the leap is anything other than a statistical artifact. Upward or downward trends in academic excellence cannot be determined from the ratings.

The second aspect of concern is that the weighting formula is not empirically derived. When deciding how much each factor will contribute to a given year’s rankings, the researchers rely on nothing more rigorous than their own opinions. Because they lack current, firsthand knowledge about the inner workings of higher educational institutions, their opinions are not, a priori, any more trustworthy than those of the average high school valedictorian.

Significantly, one factor always turns out to be the most important to the US News & World Report raters. This factor is peer assessment, and, unsurprisingly, it also always accounts for something between one-quarter and one-third of an institution’s total score—far more than any other single factor. The metaphorical heavy hitter of college ranking, peer assessment probably serves the technical function of a statistical anchor that stabilizes results against shifts in weightings or real world conditions. (Only the researchers can know for this certain, and they aren’t telling.)

To measure peer assessment, the researchers survey academic administrators about the caliber of the other institutions in their predetermined category—a strategy which, among other weaknesses, ignores the high turnover among top academic administrators. In a good year, only about half of these college presidents, deans, and the like bother to respond to the survey. Respondents are presumed to have opinions indistinguishable from non-respondents.

From an outsider’s point of view, it seems reasonable to assume that when academic administrators answer surveys, their responses reflect the priorities of their profession. Typically, these priorities center on increasing endowments and garnering support for long-term development campaigns—activities having scant impact in the here and now. Surveys of educational institutions by faculty, employers of recent graduates, or current college students would undoubtedly result in very different rankings. Are prospective students, parents and the American public sufficiently gullible to accept that college and university presidents are the best people to evaluate the relative merits of one academic community over another?

Without a doubt, making a college match is a major life event. It is a decision that shapes life choices and life chances. In our estimation, this decision is far too important to depend on dubious information gathering and clandestine mathematical manipulations. On a scale of one to one hundred, Becker Academic, LLC, gives the annual college rankings by US News & World Report a zero.