PolitiFact still biased after all these years

It's time for the annual update to our ongoing research project measuring PolitiFact's bias in its application of "Pants on Fire" ratings.

In spite of its frequent publication and pimping of "report cards" showing how various persons and organizations rate on its trademarked "Truth-O-Meter," PolitiFact openly admits that its process for choosing which claims to rate is not scientific. PolitiFact maintains it is making no effort to figure out which persons or groups lie more, albeit at the same time publishing its stories in a way that encourages readers to draw such conclusions based on unscientific data.

We at PolitiFact Bias have helped pioneer the practice of using the fact checkers' rating systems to measure ideological bias in journalism. The "Pants on Fire" bias study was the first we published. It examines differences in how PolitiFact applies its "Pants on Fire" rating compared to its other rating for false statements, "False."

PolitiFact's liberal defenders have a ready defense when it turns out that PolitiFact shows much more enthusiasm for giving "Pants on Fire" ratings to Republicans than it does for Democrats. "Republicans simply tell the biggest lies," they say, or some such.

Not so fast, we say.

One PolitiFact critic insider called PolitiFact's ratings "coin flips." Current PolitiFact editor Angie Drobnic Holan recently described the difference between a "False" rating and a "Pants on Fire" rating:

"We have definitions for all of our ratings. The definition for "False" is the statement is not accurate. The definition for "Pants on Fire" is the statement is not accurate and makes a ridiculous claim. So, we have a vote by the editors and the line between "False" and "Pants on Fire" is just, you know, sometimes we decide one way and sometimes decide the other."

If awarding a claim a "Pants on Fire" rating was based on something objective, we suggest that somebody at PolitiFact could describe how that objective process operates. We've yet to see it in over seven years of observation.

Given PolitiFact's disinclination or inability to reveal any objective basis for the distinction between "False" and "Pants on Fire" statements, we conclude that the difference between the two is substantially or perhaps entirely subjective. So we compare the percentage of all false statements ("False" plus "Pants on Fire") rated "Pants on Fire" by party. It reveals that PolitiFact, after 2007, has consistently given false statements by Republicans "Pants on Fire" ratings at a much higher rate than those from Democrats.

And that, we claim, is an objective measure of ideological bias. The "Pants on Fire" rating amounts to an opinion poll of PolitiFact journalists as to which false ratings are ridiculous. PolitiFact subjectively finds the false claims of Republicans more ridiculous than those of Democrats.

It's not the only measure of ideological bias, of course, and it is subject to some limitations that we've described in earlier publications.

The new data for 2014

Our findings for 2014 proved very consistent with PolitiFact's performance from 2008 through 2013.

The PoF bias number for 2014 was 1.95, meaning a Republican's false statement, as designated by PolitiFact, was 95 percent more likely to receive a "Pants on Fire" rating than a false statement from a Democrat. That's nearly twice as likely.

The selection bias number for 2014 was 2.56, meaning the total number of false statements, "False" and "Pants on Fire" ratings combined, was 156 percent higher for Republicans than for Democrats.

Overall, that means PolitiFact rates more Republican statements false and rates Republican statements "Pants on Fire" at a much higher rate than for Democrats.

PolitiFact continued its recent trend of finding Democrats ever more truthful. Democrats tied their all-time record with only 16 statements found false, and improved on their performance in 2013 with only two of those 16 rated "Pants on Fire."

Republicans had also shown a recent trend toward fewer false ratings since a high mark in 2011, but that reversed in 2014. Still, Republicans posted their lowest percentage of "Pants on Fire" ratings since 2008, the year for which PolitiFact won its 2009 Pulitzer Prize.

The 1.95 PoF Bias number for 2014 boosted PolitiFact's cumulative figure a few hundredths to 1.74. Since its inception, PolitiFact has been 74 percent more likely to rate a false statement as "Pants on Fire" if the claim comes from a Republican instead of a Democrat.

Note

The above figures include only elected or appointed persons or party organizations such as the Democratic National Committee. We call these "Group A" figures and consider them the most reliable group in this study for showing partisan bias at PolitiFact.

We'll soon publish our findings for PunditFact, PolitiFact's effort focused on pundits, which represents "Group B" data for purposes of our research.