Yet another story (from The Boston Globe) in which some mathematical or statistical analysis is reported with no critical evaluation. And, yet again, I register my protest.
For reference, the Educational Testing Service, which was mentioned in the original article and in the letter, operates the SAT test and evaluates the results. The scoring method for the test was recently changed, so two different methods, described as “current” and “historical”, are described in the text.
5 July 2004
Not for publication; just a plea from me to you…..
In your Sunday edition (4 Jul) you published an article in which Professor Douglas Detterman claims to have established a good correlation between scores on the Scholastic Aptituded Test (SAT) and IQ — such a good correlation, in fact, that he provided specific formulae (which you published for us) for converting between them.
I don’t doubt there is some general correlation between SAT and IQ. Unfortunately, if the formulae you published were accurate representations of what Professor Detterman provided they don’t pass the smell test.
I’ve created the following quick graphs of his formulae with commentary; the graphs speak for themselves, and had anyone on your reportorial or editorial staffs done a similar smell test they might have written more skeptically of Professor Detterman’s claims instead of merely writing about how they antagonize the interests of the Educational Testing Service (ETS) and others.
Figure 1: IQ vs. Historical combined SAT
Figure 1 shows estimated IQ vs. historical combined SAT score. Two things jump out:
- By this estimate a score of 1600 is equivalent to an IQ of about 121, which defies common sense.
- For scores above about 1350, the estimated IQ decreases for increasing SAT.
On both counts it seems clear this estimator has some severe limitations, at least when evaluating SAT scores at the top end of the distribution.
Figure 2: IQ vs. current SAT (m=math, v=verbal); ref=500
Figure 3: IQ vs. current SAT (m=math, v=verbal); ref=800
Figures 2 and 3 show estimated IQ vs. current verbal (v) and math (m) SAT scores. Since the estimator is two-dimensional it is hard to visualize, so here it is presented in one dimension by holding one score (math or verbal) constant at some reference value while varying the other. In Figure 2 the reference value is the nominal mean SAT score (500) (so it shows IQ for a math score of 500 and the full range of verbal scores, and for a verbal score of 500 and a full range of math scores). In Figure 3 the reference value is the maximum score (800).
As with the estimator for historical scores, two anomalies jump out:
- The estimated IQ hardly changes at all with verbal score — to be expected from the published formula, which assigns a weighting to verbal scores only about 3% of that for the math scores.
- The estimated IQ for a maximum SAT score — double 800s — is about 128, which is, again, not aligned with normal experience.
Again it seems clear this estimator has some severe limitations, at least when evaluating SAT scores at the top end of the distribution.
- The model published for historical SAT scores predicts a stunningly low 121 IQ for people who maxed-out the SAT (1600 combined), and for scores above 1350 or so it predicts an inverse correlation — IQ drops as SAT score goes up. These are not credible estimates.
- The model published for the current SAT scores predicts almost NO correlation between verbal scores and IQ (the graph is a horizontal line) and an IQ of about 128 for people who maxed-out the SAT, another suspiciously low result.
It seems clear at the very least that Professor Detterman’s model is not representative for scores at the top end of the distribution; and it would seem, unless verbal ability and IQ are completely unrelated, that it probably underestimates the correlation between them. On its face it would seem either that the sample sizes in his study were too small, that his analyses were inadequate, or that his claims are exaggerated. None of that was even hinted at in your story.
In short, and again, a story based on mathematical analysis got the better of your reporters and editors. Since we are coming up on another presidential campaign in which we will be buried beneath an avalanche of statistics and studies purporting to prove one thing or another that the candidates want us to believe, please please please find a way to do this better!
© Copyright 2004, 2005, Augustus P. Lowell