On Sept. 14, higher education pundits collectively heaved an exasperated sigh. It was the day U.S. News & World Report published its annual Best Colleges rankings, a point of pride for institutions sitting atop them, and a source of aggravation for those that might have fallen.

The criticism that has dogged the rankings was slightly more caustic on social media this year. Many within the sector have long declared the set of lists a flawed measure of colleges' success that is too easy to game while having an outsized influence on public opinion. The health crisis has sliced deep into campuses' revenues and some are struggling to contain coronavirus outbreaks. So few observers wanted to celebrate institutions' positions on the list, though the typical news releases crowing about them still poured in. 

"People couldn't care less about rankings right now. Don't tweet about them. Don't send press releases about them. Don't write about them," Kevin McClure, a higher education professor at the University of North Carolina Wilmington, wrote on Twitter the day this year's rankings went live.

Administrators do care about the rankings, however, even now. They distill the intricate considerations of college decision-making into an easily digestible, if faulty, format that appeals to the general public. Thus, institutions can use their positions to raise their profiles and bring in donations. That's why the methodology, and how institutions stack up, probably won't shift substantially in the aftermath of the pandemic, industry experts say. The product's popularity has persisted decades, even through national plights like the Great Recession. 

The magazine will surely need to tweak its methodology to account for certain disruptions, such as the vast contingent of colleges making admissions tests optional for students entering in fall 2021. But a key determinant of where institutions fall on the list is how much money they are able to spend on students, said John Lombardi, a former college president and now director of the Center for Measuring University Performance. 

While the pandemic won't destroy the finances of top-ranked colleges, "it will have an impact on the ability of lesser institutions to compete on the terms defined by the rankings compilers," Lombardi added.

History of the rankings

The Best Colleges guide emerged in the 1980s, forever changing how institutions competed with one another. Elite colleges have submitted false data to the publication that artificially inflated their standings. In 2007, Arizona State University tied a $60,000 bonus for its president to improvement in the rankings. And Florida in 2013 constructed a funding model that gives its public colleges extra money for performing well on measures the rankings prioritize, ultimately shaping how the state's institutions operate. 

Also driving the widespread disapproval for the rankings are the metrics the magazine selects and how inconsistent or easily manipulated they are. 

The rankings consider SAT and ACT scores, which for 2021 was 5% of the formula, down slightly from last year. Baylor University in 2008 handed out bookstore credits to first-year students who retook the SAT, and gave those who significantly raised their scores a $1,000 scholarship. Higher test scores would help the institution rise in the rankings. 

Another oft-criticized piece of the assessment, accounting for one-fifth of the formula, is a survey in which college presidents, provosts and admissions deans rate other colleges. But as Malcolm Gladwell detailed in a 2011 New Yorker essay, officials don't know the ins and outs of every college, and they very likely turned to past U.S. News rankings to make those determinations, spawning a self-perpetuating cycle. The completion rate for the survey dropped from 43% last year to about 36% for the 2021 rankings. 

Robert Morse, U.S. News' chief data strategist, stressed in a statement emailed to Education Dive that the rankings' formula emphasizes graduation and retention rates. Morse added that the lists aim to expose students and families to "the broad range of higher education options available."

Publication representatives understand colleges and students have been affected by the crisis, Morse continued, and it is "currently reviewing our strategies for future U.S. News education rankings where COVID-19 may have had an impact." The current edition of rankings used data collected prior to the pandemic. 

U.S. News makes slight alterations to the rankings formula from year to year, usually to account for pressing issues facing colleges, said Jim Jump, former president of the National Association for College Admission Counseling (NACAC) and now the director of college counseling at St. Christopher's, a private pre-K-12 school in Virginia. Student debt is even more of a concern at the moment, and U.S. News added two measures for 2021 that home in on graduates' loans. It also started to rank colleges that don't accept standardized test scores. 

But the changes are never radical, Jump said. Even when U.S. News responds to criticism and incorporates new measures, they don't dislodge universities from the top spots. 

"It's always going to be Princeton, Harvard and Yale," Jump said. 

Could the rankings change?

U.S. News will need to rethink several of its standards, given that the pandemic has upended many postsecondary norms, experts say. In addition to the many colleges going test-optional (two-thirds of all U.S. four-year colleges for the fall of 2021, by one count), alumni giving is anticipated to be shaky amid the economic downturn. It accounts for 3% of this year's rankings formula.

But a wholesale rejection of the rankings isn't likely to happen at the same scale as the test-optional movement, a campaign which would have been unheard of years ago, said Alex Bloom, director of research at consultancy EAB. The general public has recognized that requiring entrance exams presents a hardship to disadvantaged students, leading to a "crusade" against the tests, Bloom said. The public hasn't yet done that with the rankings.

Colleges at the top also have little motive to want the rankings to disappear, given the attention they garner. With metrics like "faculty resources," the rankings heavily favor affluent colleges, which will remain relatively financially steady during the pandemic. Institutions at the bottom, meanwhile, crave those accolades, though they'll probably never get them, said Colin Diver, a longtime college administrator who was president of Reed College, one of the institutions to defect from the rankings system. Institutions can, and have, declined to provide data to the magazine. But that doesn't mean they aren't included, and critics of the list have contended opting out can lower the placement of those that don't participate. 

NACAC chief executive Angel Pérez, a former admissions officer, said the rankings release day is a "noisy" one on any campus. For those that dipped in the rankings, the news would likely upset alumni, students and governing board members. 

"It's insane how the self-esteem of an institution is so deeply tied to this number," Pérez said.

The task of preparing the data for submission is also an enormous one, with "very little return," he added.

Pérez said traveling overseas enlightened him to why the public latches onto the ratings. American higher ed is particularly difficult to dissect, and he said he would meet students in China and India who would have print copies of the rankings and pass his college over if it didn't crack the top 50.

Diver, who also was a trustee of Amherst College, a prominent liberal arts school in Massachusetts, said its members debated not participating in the rankings surveys but feared that doing so would cause them to fall below their competitors.

"It was always the same," Diver said. "We said, ‘We would get killed if we dropped and Williams (College) didn't.'"

Education Dive contacted five four-year institutions and one public system that promoted their placements this year to discuss critiques and potential changes to the rankings. None provided comment by publication time. 

What improvements could be made?

Pérez said he wishes the rankings emphasized more aspects of the "public good" and placed less weight on standardized tests.

Measuring quality education is nigh-impossible and subjective, but Pérez suggested adding metrics related to civics — such as whether students vote — or graduates' success post-college, such as if they landed a job or attended graduate school. An annual college ranking published by Washington Monthly magazine focuses on such factors, including student voter statistics, as well as information about salary, job placement and graduate degree attainment. 

Most colleges already collect this information through alumni surveys, Pérez said. But he has low expectations for future formula changes: "I predict just tweaks around the edges."

More colleges may refuse to fill out the rankings surveys, but not many, Diver said. When Diver's former institution, Reed, refused to cooperate with U.S. News, some believed it tumbled in the ranking as punishment. The magazine explained in its methodology for this year's list that while it publicly notes schools that didn't fill out the survey, they "incur no direct penalty in the ranking calculations."

It also might not matter now, Diver noted, as much of the necessary information is accessible through the Common Data Set, a database run by U.S. News, the College Board and Peterson's. He predicts colleges that will abandon the rankings are ones like Reed, those "with a unique educational mission." 

But the rest, particularly those that haven't budged from the top of the list in decades?

"Well, you don't give up a game you're winning at, do you?" Diver said.