About a month and a half ago The Quixotic Man posted an article criticizing the use of the GRE in admissions decisions to graduate degree programs. I left a comment there, but recent conversations with fellow faculty members at an institution where I teach, as well as one with a NYCS board member at the Graham Priest lecture on Saturday September 11th, have spurred me to develop the comments I posted on TQM’s piece more comprehensively into my own post.
I want to start by first disclosing the following: 1) I took the GRE [the general test twice, and the psychology subject test once], 2) I have been both denied admission to graduate programs (master’s level and doctoral level) and admitted to graduate programs (master’s and doctoral level), all from roughly the same ‘caliber’ of institutions, 3) I am a graduate student of quantitative methods in education and psychology, and 4) I have taught courses in measurement and assessment in these areas (u-grad/intro level). I say all of this not to add or subtract credibility from my views, but to express that, while this article is my opinion, it is an opinion based on personal, professional and academic experience.
The GRE is comprised of three sections: quantitative reasoning, verbal reasoning and analytical writing. Scores in the verbal and quantitative section each range from 200-800 (total range 400-1600), and the writing portion ranges from 0-6. It is a standardized test, meaning the interpretation of your performance is based on the relative position of your score amongst other test takers. The average performance (the 50th percentile) in verbal is 456, in quantitative 590 and 3.8 for the analytic writing section. A common understanding for grad school applicants (based on experience) is you want to obtain no lower than a 1200 combined, and depending on your intended area of study, you want at least a 650 in quantitative and a 550 in verbal.
Now, I have written before about validity (#1, #2) and properly operationalizing one’s construct of interest. What the GRE claims to measure is an overall competency as a demonstration of readiness to graduate school. But the thing is, I’ve been in grad school now for over 4 years. I’ve completed 2 master’s degrees and just began my doctoral studies. While a broad vocabulary (an important skill for high performance on the verbal section) is impressive, and might be helpful in writing papers, it does not equate to success in grad school. And while the basic math skills the GRE measures are important to have in general, depending upon one’s major, most of these never come up. And grad programs that involve mathematics more heavily, first would require a greater proficiency then that which is measured by the quantitative section of this exam, and second could be evaluated more effectively by college transcripts.
Being successful in a graduate program, in my opinion, is dependent upon psychological factors (like willingness to take initiative and work independently), and skills like critical and reflective thinking. These characteristics can be evaluated much more so in a student’s performance in u/grad via transcripts and letters of recommendation, and high performance on the GRE does not guarantee these traits are present.
However, all of this has little to do with why the GRE is probably a poor measure to rely on (I didn’t say throw it out entirely, but singling it out as a very important factor might be inadvisable). The main reason why I posit that the GRE is an instrument that might not necessarily measure what it claims to measure has more to do with the effectiveness of GRE test prep programs at eliciting higher scores.
If the GRE were measuring some stable, overall characteristic of a person’s readiness to pursue graduate studies, then why would a program geared towards helping you prepare for it be so successful at elevating scores? Without naming specific names, several programs offer test preparation services, including study material for single item purchase, up to enrollment in weeks and months long courses. These services range from $400 up through $2000. Most of these services offer participants ample opportunity to take practice tests, and teach test-taking skills.
Here’s the thing about construct validity and reliability – if an instrument is truly measuring what it claims to be, measurements ought to be consistent. If a GRE score can jump 100 or 200 points, I’d be wary that the instrument is truly measuring a stable quality of one’s abilities. If the only factor introduced in between test administrations was a test preparatory course, the test in question might be more so measuring one’s ability to take tests. As a side note, the ability to perform well on timed exams is a less than important skill in grad school.
Some might then say, OK, then take a test prep course and one would be fine. Here’s my problem with that: they cost a lot of money. Factor that in with application fees and the cost of taking the exam itself, the system begins to favor individuals with the financial means to cover such costs. This goes against the spirit of standardized tests. Long ago, in a climate when only the children of the more elite class were granted access to higher education, standardized testing came about to level the playing field. Opportunities for higher education could be offered on merit and ability, and less so on wealth and legacy. However, if the exams are as vulnerable as they seem to be by extensive preparation courses, and these courses cost sufficiently high enough fees, this begins to put the proverbial playing field into disequilibrium.
When a test proposes to measure basic capabilities as a predictive model of future success in an arena, and decisions are made granted (or restricting) access based on the results, the test itself should demonstrate validity. If the scores on this test, however, covary significantly with the degree of investment made into test preparatory courses and materials, there could be a problem with the stability of the instrument, and subsequently the conclusions drawn from the results.
