Statistics can't prove that one group is better than another.

Beware statistics that ‘prove’ one group is inherently better than another.

With the U.S. presidential election only months away, there’s a lot of talk about the role that media organisations play in shaping voter attitudes. In these discussions, one sometimes hears that Fox News viewers have been proven to be the least informed of any news consumer. This relates to a study conducted by Fairleigh Dickinson University in 2012, where participants were asked about which news sources they consumed, and were then asked 5 U.S.-focused questions and 4 international-focused questions. The authors found that participants who consumed only Fox News were able to correctly answer less questions than any other news consumer, thereby making them – according to the survey authors - the least informed. NPR listeners were able to answer the most questions, thereby making them the most informed.

A claim to be able to represent how ‘informed’ a person is based on a 9-question survey should strike any data professional as dubious. The term ‘informed’ is very broad, certainly too broad to be represented by 9 (9!) questions. Furthermore, the questions applied in the study focused specifically on current events. There were no questions relating to agriculture, infrastructure, military, education etc. All the results of this study tell us is that Fox News viewers may be less able to answer an extremely narrow set of questions. It should also be noted that the difference is not particularly large for either the international or domestic questions. There is a difference of 0.81 between NPR consumers and Fox consumers for international questions, and a difference of 0.47 for domestic questions. These figures indicate that Fox News viewers can answer, at worst, one less question on average than NPR listeners.

The Fairleigh Dickinson study bears a striking similarity to a study by Satoshi Kanazawa that claimed to ‘prove’ that liberal voters were more intelligent than conservative voters. But once again, the methods don’t stand up under scrutiny. As Shawn T. Smith writes, Kanazawa’s reasoning suffers from two key flaws:

1.     Participants’ self-identification of political preferences. Participants were asked their political preferences when they were aged 18-28, which is the age when people are most likely to say they are liberal.

2.     A highly incomplete measurement of intelligence. Intelligence is determined through a test where a participant is shown a picture and asked to name synonyms for that picture. At best, this could give an indication of how an individual would perform on a verbal test.

The points I’ve discussed above deal specifically with experiment methodology and interpretation of statistics but there’s a more basic principle at play: Beware statistics that purport to show that one group is inherently better than another. Beware also statistics that purport to prove things with respect to intangible qualities. We have intuitive understandings of terms like ‘intelligence’ and ‘informed’ but they have no universally accepted definition. Studies that make strong statements about these terms tend to use a narrow definition that most favours their work.

No group has a monopoly on humanity’s positive traits. Be wary of any statistical analysis that purports to show otherwise.