Facing the Test: Grad School as Statistical Uncertainty

It has been an unusually trying year for students applying to graduate schools of medicine, law and business. And for the makers of the standardized tests required of all college and grad school applicants, the year has been a real stinker. Errors in translating raw test scores into the composite scores sent to school admissions offices marred both the Medical College Admissions Test (MCAT) and the Graduate Management Aptitude Test (GMAT) this year, while alterations in the content of the Law School Admissions Test (LSAT) seriously skewed the results and forced some law schools to reevaluate students whom they had admitted before learning of the shift.

Standardized tests have also recently become a focus of attack in Congress. A bill now before the House of Representatives may force major changes in the content of standardized tests and limit the role the tests play in the admissions process. And the Bakke case, now before the Supreme Court, challenges the legitimacy of these tests altogether, as proponents of affirmative action charge that the tests are culturally biased.

The American College Testing program (ACT), the organization that administers the MCAT, wrote to medical schools in May to inform them that ACT had made an "equating error" in scoring the April MCATs, and that approximately 90 per cent of the students tested received scores that were slightly low. Apparently, ACT's error will not affect the admissions process. According to Charles Sentress, coordinator of public affairs for ACT, medical schools should receive corrected test scores some time this week. Since most students who took the April MCATs are applying for admission to the class that matriculates in 1979, at the earliest, medical schools should have ample time to evaluate students in light of the corrected scores.

A similar "equating" error in the November GMATs had only a mild effect on the business school admissions process for the fall of 1980. In late November, ETS notified business schools of the error--a miscalculation of about ten points on an 800 scale--and by January had sent corrected scores to the test-takers as well as the schools. Amy Meyer, associate director of admissions to Harvard Business School's masters program, says that because the error was so small, and because the Business School received prompt notification about it, the effect on admissions was minimal. "There was a group of a little over 100 people on whom we had already made a decision, but when we discovered the error, we reviewed all those applicants," Meyer says adding, "We did not change any decisions. GMATs count for approximately one-sixth of the decision so a nine or ten point error is not very significant."

The difficulties surrounding this year's LSATs, however, clouded by the obfuscations of ETS spokesmen, may not be quite so insignificant or so innocuous. Between the July and the October testings last year, ETS introduced substantial changes in the LSATs. As a result, about three times as many law school applicants as in the past received perfect 800 scores, while about twice as many scored in the 750 range. ETS neglected to mention the changes to law schools until this past winter, when a number of law schools jointly asked ETS about the unusually high scores. As a result, some law schools accepted students based on the erroneous assumption that the post-July tests were no different than earlier LSATs.

James W. Zirkle, associate dean of admissions at Yale School of Law, says his office received notice of the error "early enough to go back and reevaluate people with abnormally high scores." Other law schools--particularly those that use a "rolling admissions" process--were more affected by the skewing. Many of these institutions, including Harvard Law School, did not reevaluate applicants who had been admitted before the schools discovered that the scores were skewed. Patricia Lydon, dean of admissions at the Law School, says the reason her office did not re-examine early admittees was that in general "the ones we decide on early are very strong all around."

Steven Brill, a graduate of Yale Law school and a regular columnist on the law with Esquire Fortnightly, said he thought the change in LSATs had its greatest effect on law schools with rolling admissions. "Rolling admissions law schools take people at the top of the curve first, which means that they take the guy with the 800s first. If a guy who took the test in July gets placed in a holding file with a 720 score, and the law school meanwhile accepts people with higher scores from the October LSATs, the 720s guy may get lost in the shuffle," Brill said.

The way in which ACT introduced major changes into the MCATs two years ago provides an interesting comparison with ETS's methodology. Five years before the change, ACT began testing experimental questions in preparation. "This was a long, gradual process, and the med schools knew about it from the start," Sentress, of ACT, said.

Where the change in the prospective lawyers' test involved an increase in math questions, the revised MCATs introduced a larger number of science questions--to test for both specific knowledge useful in medical school, and for the ability to think in a scientific mode, Sentress said.

John P. Smith, media director for ETS, said last week the purpose of the change in the LSATs was "to give the better students a chance to excel, by sprinkling rougher questions throughout the upper range of the LSATs." Before ETS altered the test, Lydon explains, "there was a lumping of people scoring in the upper-600s, and an extreme drop in the over-700 scores. What they did was to add more difficult questions, in order to make the scoring curve more like a normal bell-curve distribution."

Smith's explanation for ETS's failure to notify law schools of the change sooner is somewhat disturbing. "We honestly did not recognize that the change would produce this result. It's not that we were hiding the situation," Smith said, adding that ETS sent out "interpretive data" as soon as it recognized the error.

His explanation suggests atypical carelessness on the part of ETS testers. Within three days of the release of the November GMATs, for example, ETS officials had discovered the minor calculating error and immediately notified business schools. But with the LSATs--the scores of which were far more skewed than were the GMATs--ETS took about two months to "discover" the error, and another month to issue guidelines for interpreting scores.

Another murky side of this past year's LSATs was the nature of the changes made. Cathleen Jolly, a spokesman for ETS, claims ETS designed the recent exams to be more difficult than the past LSATs in order to "give more opportunity to the students with higher ability." But Brill says the changes ETS instituted make the test harder only for some applicants. "ETS added more math, figuring that would make the test harder, which it did," Brill says. "But the people like me--people who are strong in math and who do well on standardized tests to begin with--they only do better." If Brill is correct, it appears that the change in the LSATs does not necessarily allow the more able potential lawyers to excel, but rather merely extends the advantage enjoyed by those more proficient at math and at test-taking. Peter Liacouras, dean of Temple University's law school, told a Wall Street Journal reporter in February that the LSATs, even before the changes, failed to measure "common sense, motivation, judgement, idealism, client-handling ability, oral skills and leadership," among other skills.

Harvard seniors who applied to law schools this year appear to be less upset with the nature of the change than with ETS's failure to notify law schools sooner. Linda Novak '78, who took the LSATs before the modification and received a score of 730, says she thought the skewing had probably hurt her chances for admission. "I was rejected from Harvard Law School, and I had really thought I had a good chance of getting in," Novak says, adding that the admissions office was right to stand by the students it had accepted before receiving ETS's notification. "Once you admit somebody I don't think you have a right to withdraw that admission, but I do think Harvard should have done more with the people who may have been unfairly rejected."

Joseph Savage '78, who was not admitted to Harvard Law School after having been placed in the hold category, says he does not believe the LSATs played a major role in his rejection. "The people Harvard rejected from the hold category weren't the ones who got screwed. The people who were rejected early on, before the Law School knew about the error--they're the ones who got screwed," Savage says. He adds, "Standardized testing ought to be reviewed in general, I think, not only in the light of the LSAT screwup, but the MCATs, too."

Savage is not alone in this point of view. As ETS continues to grow, and its "surplus funds" (it enjoys non-profit status) expand beyond the million-dollar mark, the legitimacy of standardized tests is being challenged from a variety of sources. The Bakke case raises serious questions about standardized tests--regarding the kind of information they reveal about a student and the possibility that the tests are culturally biased. The very existence of special admissions programs like the one at the University of California at Davis Medical School, from which Bakke was twice rejected, is based in part on a belief that standardized tests are an inadequate means of evaluating applicants.

Rep. Michael Harrington (D-Mass.) may pose an even more direct threat to the standardized testing establishment. He is currently rewriting a bill he introduced in the House last year seeking to limit the use and power of standardized tests. "The basic aim of this bill is to see a less mechanized, less fallible process," James Castello, Harrington's legislative assistant, says. "What you can read from someone filling in little IBM bubbles for three hours does not amount to much," he adds.

In its re-written form, the bill will probably stipulate that colleges and graduate schools must stop using "cut-off scores" on tests as the basis for either automatic rejection or automatic admission. Another stipulation would prevent school admissions officers from discriminating against students who choose not to take standardized tests.

If it passes, Harrington's bill will force schools to develop more thorough, accurate admissions procedures. Castello says he thinks schools tend to place too much emphasis on test scores because of their convenience. "ETS will tell you that a statistical variation of 60 to 70 points is insignificant," Castello says, adding, "That range of error tends not to be widely enough respected. A law school regards an applicant with 700s very differently from one with 630s, for example."

Castello admits, however, that the bill "has not exactly caught on like wildfire," and that it is not likely to pass in the House. Despite the dark cloud under which the testing establishment has fallen recently, the immense market for standardized tests--such as the minimal competency tests that many states are now adopting--is growing, Castello says, adding, "As a nation we are showing no sign of decreasing our reliance on these tests."

For the near future, at any rate, it appears standardized tests will remain an integral part of the application process in America, and an unpleasant part of our lives. One Harvard senior, who is still on the waiting list for admission to the Law School, said he feels "frustrated to think that I might not get into a school because the school accepted an abundance of people who scored better than me because ETS fucked up. It's one thing to be beaten out of a fair game, but this is harder to take."