PunditFact's "Pants on Fire" bias, 2014

We pledged to apply our "Pants on Fire" bias research methods to PolitiFact's "PunditFact" project.

PunditFact is the branch of PolitiFact that looks at and rates statements from pundits. PunditFact uses the same cheesy "Truth-O-Meter" system to which PolitiFact has wedded itself.

Our "Pants on Fire" research project looks at how PolitiFact disproportionately applies its "Pants on Fire" rating.

It's important to note that we don't simply look at the higher numbers of "Pants on Fire" ratings PolitiFact gives to Republicans and conservatives. Nor do we focus on which party receives more "False" ratings. We note that PolitiFact has never provided anything resembling an objective criterion for distinguishing between "False" and "Pants on Fire" ratings. We conclude from the evidence that PolitiFact very probably distinguishes between the two ratings subjectively. From that, we conclude that the proportion of "Pants on Fire" ratings in relation to the total number of false ("False" plus "Pants on Fire") ratings tells us something about the ideology of the people applying the ratings.

PunditFact shows a remarkable skew to the left.

PunditFact's PoF Bias number for 2014 came in at 2.19. That simply means PunditFact was 119 percent more likely to rate a conservative's false statement "Pants on Fire" than a liberal's.

Adding the small amount of data from the end of 2013, when PunditFact was first starting out, we obtain a cumulative figure of 2.57 for the PoF Bias number. So over PunditFact's entire lifespan, conservatives were 157 percent more likely than liberals to have a false statement rated "Pants on Fire."

We also have a hint in our data that PunditFact shows much more interest in statements coming from conservatives.

Of course we're only looking at two categories of statements, so we place no great importance on that aspect of the chart. We note, however, that PolitiFact's oft-stated criterion for choosing its subject matter ("Is that true?") fits just as well (if not better) for the hypothesis of liberal media bias than for the idea that conservatives simply lie more.

It's more natural to question statements that do not fit with what one accepts as true, after all.



Correction Jan 19, 2015: Revised 7th graf to correct description of cumulative totals.