The Impact of
US News and World Report College Rankings on
US News and World Report College
Rankings "us news" best college, liberal arts, school, rankings
Recognizing The
Deficiencies Of The US News & World Report College Rankings
The University of Southern Mississippi as a case example
October 7, 2004
This paper is considered to be a sphere of silence
article.
--- The greatest tragedy of this list is that it is marketed to students, but
no feedback from students is used to compile the rankings.
top rated colleges?
rankings controversy
world report, college and school tiers
Introduction
If students can be ranked by school officials—and they most certainly are—then schools can be ranked too. Time, Princeton Review, Money Magazine, Newsweek, and Peterson’s Guide are publications in which some type of school status is reported to the general public. None of those are as popular as the list of college and university rankings printed by US News and World Report since 1983. The rankings in this magazine are actually part of a larger, more in-depth guide to going to college called “America’s Best Colleges.” Every edition of these rankings is made controversial by the schools not in the top two of its four levels and especially by those schools who are not on the list at all.
There is little doubt that, depending on who is reading the list and why they are reading it, the controversy of these rankings escalate the hype that editors often need to help sell their publications. However, these rankings would not be so contentious if readers would consider the obscure intricacies of “America’s Best Colleges.”
Usefulness Of “America’s Best Colleges”
There are positive aspects of the guide. To begin with, being in just one of the possible four tiers should be an honor regardless of placement because far more schools did not make it on the list than those who did. Most of the schools in the top two tiers are private schools, so if a public university makes it onto the bottom levels, this too should be viewed as a positive circumstance. According to the magazine, there are 217 liberal arts colleges. Arguably there are at least 800 liberal arts colleges in America. Although many people will label most of those colleges as vocational schools instead, not even half of those are on the list. Moreover, it should not be forgotten that US News and World Report also refers to the schools at the bottom of the list as part of “the best colleges.” Indeed the title of the whole list is “America’s Best Colleges.”
The guide also serves as a beginning for new students who are seeking a non-local school and who are not held captive by their hometown, family, or jobs. Useful financial information for students is provided in the guide. For example, both parents and students will appreciate knowing that the average debt of a University of Southern Mississippi student after graduation is $11,000. The debt of a Virginia Tech student, as a comparison, will be about $25,500. Money is more often than not a deciding factor on which school to attend, and this subject is covered well in the online version of the list. The guide also has a very good directory of both public and private schools, arranged by state, that includes contact information, tuition, and student population. Unfortunately, the problems of the list overshadow its contributions and advantages.
Inaccurate, Misleading, or Inutile Rankings
There are items important to many students that are not shown by the rankings, such as the city environment in which the school is located; the chances of finding employment upon graduation; the number of online classes available; the number of night classes available; the available fellowships and scholarships and the number of students that are getting them; having to use a shuttle to get from one building to another between classes on a very large campus; the comfort of the classrooms; and last but certainly not least—inspirational and dedicated professors. In all fairness though, all these questions can be answered by contacting a school directly, which is facilitated by the school directory provided in “America’s Best Colleges.” What will be difficult to obtain, however, is a useful interpretation of the scores and an explanation of the pitfalls of the scores used to compile the list.
The lay parent—and definitely the student—will not correctly understand the true difference between number 1 and number 10 or even tier 3 and tier 4 on the list. They will not understand the importance of a number being statistically significant. A familiar example can be used as a mental picture of this point. When one student makes an A on a test while another makes a B, there are so many possible reasons why the student made a B instead of an A that it is difficult to believe the A-student is smarter than the B-student. Likewise, is a tier 3 school really better than a tier 4 school because it appears higher on the list? Perhaps what readers of “America’s Best Colleges” do not know will not hurt them though. Afterall, the producers of the list make absolutely no use of the statistically significant gauge. For a further example of why results need to be assessed for statistical significance, the reader should see Appendix A. Appendix A is a comparison of scores from a tier 2 and tier 4 school, which happen to be very similar in spite of the difference in tiers.
Even when percentages are used to show a comparison, many of the percentages are not all that valuable to students. At least 20% of the overall calculation for a tier score comes from how many freshmen the school has and how many of those return the following year and eventually graduate. Another 5% comes from how many alumni donate to their schools, though no consideration is given to how large or small the donations are. College students, who work at least part-time for their four to six years of school, will candidly say that this 25% total is worthless to them. More valuable factors for the students and their parents are the student-faculty ratio and the proportion of full-time faculty. Yet, these two characteristics constitute only 2% of the total score used to rank the schools.
The producers of the list also emphasize the rankings that incoming college students had from when they were in high school, which is referred to as Freshmen in the Top 10% of High School. A separate paper could be written about the shortcomings of just this one measure. It is well-known that high schools are not created equally. For example, students in Ocean Springs, Mississippi have continually scored higher on the ACT than students across the state. Schools in this city have a much lower drop-out rate than other schools. Yet, at impoverished schools where teachers have to be given extra incentives to work, students are given the same value in US News so long as the students are in the top ten percent.
Furthermore, schools accepting lower ranked students can actually be penalized on this list. One problem with this is that a large number of beginning college students are in their middle twenties. A 24-year-old student might not be so marginal as he or she was at 18 years old. Should a school that decides to accept more marginal students be stigmatized in light of how important education is in American society? Alot can change both mentally and academically by the time a college student goes from 18 to 24 years old. Parents paying thousands of dollars for their children’s education might like to know what they will be getting in return. A greater emphasis, therefore, should be placed on the quality of students after graduation rather than before, but that is not done in “America’s Best Colleges.”
Yet another problem with the scores is that the data used to compile them have a great chance of being miscalculated or deliberately misrepresented. Some school officials might inflate or otherwise exaggerate the admission policies, retention records, or alumni records of their schools in the hopes of maintaining or changing a spot on the list. What happens in the statistics when one school official leaves out figures for students admitted on a probationary status but another school includes them? What happens when one school includes test scores for international students but another school does not? What happens if numbers for early-decision high school students are used in the scores, but many of those students decide to go elsewhere? These hypothetical questions could go on and on, but no disclaimer to offset them is provided to the readers of “America’s Best Colleges.” In short, non-informed readers are misled.
Readers of “America’s Best Colleges” are told that research “shows that the more satisfied students are about their contact with professors, the more they will learn and the more likely it is they will graduate.” Yet there is no assessment of how likely a student is to get a graduate assistant who lectures and grades papers in lieu of the professor who needs to spend more time doing research and writing. Equally perplexing is that there is no attempt to survey any students. Because there is no survey of students who attended the schools on the list, there is no examination of how programs help students develop interpersonal skills or the ability to integrate and assimilate multiple kinds of information. Furthermore, there is no attention to the ever-increasing importance of life-long learning.
Even the five factors used to evaluate the commitment to instruction of some schools are not good predictors. These factors are class size, faculty salary, professors with terminal degrees, student-faculty ratio, and proportion of full-time faculty. None of these factors are widely used by students in making the most important of decisions: which school to attend. Being realistic, just how many parents look for these items when they are selecting a school for their students? All of these negative aspects contribute to “America’s Best Colleges” being either misleading or inutile.
Why Does A School Drop Tiers?
Whenever a school rises or falls considerably in rank, the greatest of all questions should be asked: why? The top 15 schools never seem to change now, so it might be that changes will occur at the lower levels instead. Top-level schools appearing earlier in the publication did change rankings easily at one time though. For example, Reed College was once in the top ten of “America’s Best Colleges.” When the president of Reed refused to participate in the surveys used to compile the list, Reed College suddenly plummeted to the bottom. Just as suspicions were raised when California Technical University went from number 9 to number 1 a few years ago, so too should skepticism be given to the fall of the University of Southern Mississippi from tier 3 to tier 4. The fact is that the educational data used to compile “America’s Best Colleges” do not fluctuate from year to year as does the methodology and dependability of this list. The University of Tulsa in tier 2 was in position 93 last year but it is 96 this year. Its overall percentages changed very little between the years. In fact, the largest percentage change for the score of any item on the list was just 3%. Its overall score dropped only 1 point. Across the board, scores for the University of Tulsa and the University of Southern Mississippi are quite similar, but the former is in tier 2 and the latter is in tier 4. For a chart comparison, see Appendix A.
What exactly occurred differently at USM from 2003-04 that resulted in its being relegated to tier 4 in 2005? Exactly why did USM fall one level in rank when its enrollment and retention went up; its acceptance rate is 27% lower than the University of Tulsa, and its peer assessment score is only four tenths of a point lower than that same tier 2 school? Those in charge of the rankings will say that changes in scores of other schools are the likely cause for the movement of another school. However, there were three specific scores that helped bring USM down.
Two parts of the overall scores were not completed by the administrative officials who were in charge of this information at USM. These portions were the freshmen who graduated in the top ten percent of their high school classes (6% of the total score) and the alumni giving rate (5% of the total score). Being left unanswered, these two items accounted for 11% of the total score. Every school in tier 1, tier 2, and tier 3 had this information completed. USM did not. Why?
The university president at USM was not in complete charge of compiling data for the current ranking. Obviously the president would not have deliberately left these questions unanswered. Some responsibility for these items being left unanswered, which accounted for 11% of the total score, must lie with USM officials beneath the current university president. One USM faculty member with a good deal of responsibility over the data resigned as the rankings were being released. Still, one must wonder why the evaluators of “America’s Best Colleges” did not use previous data for these scores. They extended that courtesy to four other institutions that did not present information for these items, and they used alternative sources to get the data for three other schools. Values for these seven schools were not marked unavailable, which is referred to as “Not Available” on the list. Curiously, the scores for USM for unanswered data were not listed as unavailable and USM did not have its missing data filled in even though data from the previous year was available. Yet, these seven other schools were given that treatment. Is this list really arbitrary as some critics have claimed? If these values were filled in, especially with the data from the previous year, USM would have remained in tier 3. Unfortunately, even if the figures for USM had been included, USM would not have risen any higher than tier 3. This is because of its peer assessment score.
The peer assessment score of USM was a large factor in the score of USM. Accounting for 25% of the total score, peer assessment is based on surveys. A comparison of the figures for peer assessment of another school is enlightening. With a score of 5 being the highest, Harvard University in tier 1 holds a 4.9. The University of Tulsa has across-the-board scores similar to those of USM but it is still ranked two tiers higher than USM. Its peer assessment value is 2.6. Peer assessment for USM is listed at 2.2. It is easy to think that peer assessment, as used in “America’s Best Colleges,” has hurt the overall ranking of USM. Interestingly, the peer assessment score for USM at 2.2 is the same as it was last year! If this 25% of the total score was unchanged, should USM have fallen from tier 3 to tier 4? There were 2,457 academic presidents, provosts, and deans who served as the only survey respondents that assessed the value of a certain program within a college. One should question how honest, accurate, and neutral these survey responses can be. Were there other factors that made USM fall?
Peer assessment and other items on “America’s Best Colleges” can easily be adulterated by people with ulterior motives. At USM, for example, some faculty were resolute in removing the university president from office for various political reasons. They conducted an extensive letter writing campaign that reached university administrators and editors of academic news media across the country. This handful of people tried for two weeks in the local news to convince the public that the president alone was responsible for the tier 4 status of USM and that the fall in tiers was opprobrium. In fact, the American Association of University Professors (AAUP) is listed by USN&WR as an overall source for some of the missing data compiled in the rankings. With the USM chapter of the AAUP so heavily involved in removing the university president from office, the AAUP should not be relied upon at all. All of this in and of itself is demonstrative of the deficiencies that can pervade the rankings published by US News and World Report. All of this in and of itself shows that the list can be inaccurate and unfair.
What Can Low Tier Schools Do?
The schools in the lower levels could stop reporting their data to US News and World Report, as some have, in order to stop the publication of its rankings or to effect other changes in its ranking methodology. Officials from those schools could also band together to create their own list. Mississippi, New Mexico, Alaska, Idaho, Montana, North Dakota, South Dakota, Maine, Nevada, West Virginia, Wyoming, Hawaii—these twelve states do not have schools above tier 3. Some appear only in tier 4. A few more states have no public schools in the top two tiers, and the overwhelming majority of schools in tier 1 are private schools. These schools could contact USA Today or some other national print medium to entertain a partnership. Even creating a website—administered by a neutral party—could eventually serve as competition to the USN&WR guide, especially if it is free, in-depth, addresses the needs and interests of both the parent and student, and includes input from current and graduated students.
However, the above suggestions more than likely will result in the producers of “America’s Best Colleges” finding alternative sources for their data, such as the National Collegiate Athletic Association and the American Association of University Professors. That widens the avenue to biases and misrepresentation. The producers of the list already rely on these two associations for data more than they should.
Sadly, “America’s Best Colleges” is so popular that probably no action of school officials will stop its publication. A better action would be to work with the producers of the list to get them to incorporate into the list more accurate and definitive measures of academic value. It would be good to have a small disclaimer included about how near impossible it is to measure student quality. Also, this list would not be controversial at all if the producers could be convinced to de-emphasize the ordering of schools by numbers in favor of an alphabetical listing. Going to two tiers in lieu of four would also decrease the criticism from many school officials. That is, combining tiers 1 and 2 into a level 1 and combining tiers 3 and 4 into a level 2 spreads the glory and infamy a little more fairly.
Pressuring the editors of USN&WR has caused a few changes in the methodology and format of the rankings. The producers of the list have created more categories so that more schools can be included at least somewhere on the list. They should be pressured into creating even more categories that would address specific programs. For examples, USM has a well-known Marine Science program and a nationally known Polymer Science program. That information is not found in the rankings though. Xavier University in Louisiana, although probably due to race and sex, once sent more students to medical school than any other school. That is not known in the rankings either. Categories have already been created for business and engineering programs, so additional categories reflecting other distinctions would be useful to readers and perhaps helpful for low-tier schools.
The tier 4 schools could advertise in the guide, as do some of the other lower schools. USM had a full-page advertisement in the 2003 issue, although there is not one in the current issue. Perhaps some parents and students will become interested in a school by its advertising instead of its rankings? There is nothing wrong with that. At a cost of about $20,000 for a full-page ad though, advertising in this instance might not be cost effective.
Conclusion
Unfortunately, so
long as alumni contributions, faculty salary, school funding, and ACT scores are
part of the criteria to compile the infamous rankings, USM will not go higher
than tier 3. A lot of that has to do with state politics and is no fault of the
school. How many states, for instance, have a supreme court that forcefully
lowers college admission requirements like the Mississippi Supreme Court did in
the infamous Ayers case? Furthermore, so long as the producers of the
rankings seek outside sources for their information on USM and make peer
assessment account for 25% of the total score, the rankings will never improve
past tier 3. The sad circumstance is that due to the popularity of “America’s
Best Colleges,” USM and other low-tier schools must still strive to do well on
this list . . . no matter how inaccurate, misleading, and inutile the list is.
Appendix A
Comparison Of Tier 2 School To Tier 4 School With Similar Scores
For
interpretation, it should be remembered that each column counts as a
different and separate percentage of the overall score. Columns A, B, G, H, and
K--with just a few points difference--account for 60% of the overall score.
Column J, with a lower number being better, offsets Column C, in which Tulsa
does better. Column I, with its shortcomings, was explained earlier in
this paper. Yet, one school is in tier 2 while the other is in tier 4.
A B C D E F G H I J K
Tier 2 Tulsa 2.6 78 70 55 62 1 89 22-29 60 76 20
Tier 4 Southern Miss 2.21 72 52 46 47 10 90 18-24 NA2 493 NA4
Legend:
A = Peer Assessment; B = Retention Rate; C = Predicted Graduation; D = Actual Graduation; E = Class % Under 20; F = Class % Over 50; G = Faculty % Full-Time; H = ACT 25-75 percentile; I = High School Students in Top 10% of class; J = Acceptance Rate; K = Alumni Giving Rate
1. Accounts for 25% of the total score, with 5 being the highest.
2. Accounts for 6% of the total score. The figure for the previous year was 36%.
3. A lower number here is actually better because it indicates the admission process is more competitive. For example, Harvard’s 2002 figure was 11%.
4. Accounts for 5% of the total score. If data from the previous year had been used, the score for USM would have been only 2% less than the score for Tulsa.
**Note on sources for peer assessment in the section of this paper titled Why Does A School Drop Tiers. Some of the information and opinions in this section of the paper were derived and formed from the following sources: an interview with one person in the USM president’s administration; conversations with two USM professors in the Arts & Sciences; a six-month collection of news articles and opinion-editorials in two local newspapers that covered the president-faculty controversy at USM; several editorials in the Chronicle of Higher Education; a public termination hearing of two USM professors; one public statement from the USM Alumni Association; and one public statement from the president of the USM Alumni Association; and several email sent through the USM campus-wide email system that had been arranged by some of the faculty. This author witnessed first-hand the public appeal of some faculty members to the local TV media and of several faculty members who effected a public appeal through the print news media.
USN&WR was contacted as to why previous data were not used for USM in place of the missing data that were needed for the rankings. The editors were specifically asked why they used other data sources for seven other schools but not for USM. After waiting two months with no reply, the basic feedback form on the company's website was submitted. An email from the Director of Editorial and Reader Services came stating that the request had to be "on school letterhead and signed by the department head." So another request to a different person was sent to the magazine through post office mail (04-13-05), and within two days USN&WR sent another reply email stating that the department head had to sign the request. At this time, no direct answer is expected from USN&WR on this matter.