PolitiFact, percent error, partisanship

In last Sunday's post about PolitiFact's math on uninsured Americans, we mentioned the worth of tracking PolitiFact's application of a basic math equation used to calculate error percentages.

To calculate the percent error, one takes the difference between the right figure and the wrong figure, and divides the difference by the right figure.

For example, if we guess Scarlett Johansson weighs 175 lbs and her actual weight is 100 lbs, we take the difference, 75, and divide it by the correct figure, 100.  That gives us 75 percent, so that's the percentage error.  It is incorrect to divide by the incorrect estimate, 175 lbs.  That procedure ends up giving us a figure of about 43 percent, which is the wrong figure for the percentage error even if the calculation is performed correctly.  It's the wrong formula.  Our estimate was off by 75 percent, not 43 percent.

We propose, given the difficulty journalists often display with math problems, that this calculation can serve as an indicator of ideological bias.  Where a journalist performs the wrong equation to the benefit of one political party over the other, we have an evidence of ideological bias.

The Experiment

We used the search string "off by" and the term "percentage" and performed a Google search limited to the politifact.com domain.  We then combed the results for the target results: instances of percentage error calculations.

We found only 14, which is a low number for a study.  Two of the fact checks had two (wrong) calculations in them, but we count each fact check as just one case.

Out of 14 equations, PolitiFact performed the wrong calculation an appalling nine times.  Over half.

Good Fortune for Democrats

Since there are two ways to perform the percentage error calculation, albeit one is the wrong way, we tracked both methods for their correlation with benefit or harm to political parties.  The reason is simple.  One of the calculations will normally minimize the error compared to the other calculation.  If Republicans get the wrong calculation every time the wrong calculation minimizes the error and the right calculation every time it likewise minimizes the error then just counting the number of times the wrong formula helped Republicans doesn't adequately tell the story.  The other half of the story is the uncanny ability to use the right equation when it helps Republicans.

Does anybody sincerely expect PolitiFact to help Republicans?  Do you work for Media Matters or what?

Wrong calculation helped Dem., harmed Rep.	6
Wrong calculation helped Rep., harmed Dem.	3
Right calculation helped Dem, harmed Rep.	3
Right calculation helped Rep., harmed Dem.	0

Each of the nine times PolitiFact used the percentage error concept for a fact check of a Democrat in our sample, the Democrat received the calculation that provided the greatest benefit.  That's not counting the two ambiguous cases. [incorrectly phrased; ignore--ed.]

Where the type of calculation made a noticeable difference in the degree of error, the liberal point of view got the benefit 75 percent of the time.

Three times PolitiFact aided Republicans with the wrong calculation.  One case occurred in a fact check by an intern.  Another case helped out Republican Allen West by a measly 2 percentage points.  The third case helped a Republican criticizing the influence of a special interest group on an election.  In other words, the truth is even worse than the table makes it look.


Correction 5/29/2014:  Allen West received the benefit of two measly percentage points, not Herman Cain as I originally wrote.  Cain occurs in our data as a case of harm from the wrong calculation.


Data notes


My spreadsheet is a little chaotic, since I used it to make a few notes on potential PolitiMath cases.  Look for red or blue boxes in the harm and benefit boxes to find the fact checks that make up our sample.

We believe our search method adequately simulates random selection.

Who made the mistakes?  The list of writers and editors on the flawed stories reads like a Who's Who of PolitiFact's central movers and shakers, including Bill Adair, Angie Drobnic Holan and Louis Jacobson.  Former prolific PolitiFacter Robert Farley's also on the list.  Farley now works for FactCheck.org.

Additional note

Congratulations to Jon Greenberg, writer for PolitiFact's PunditFact project.  It didn't turn up through the search we designed for our study, but Greenberg used the right calculation for his fact check of Fox News' Kimberly Guilfoyle last week.

If Greenberg had used the wrong formula, Eric Bolling's percentage error would have shrunk from 50 percent down to 35 percent.  This would have counted as a benefit to Republicans on our spreadsheet.
Greenberg's fact check harms Republicans more than it would have if he had used the wrong calculation.  The second "3" on our chart would change to a "4" with this stat included.


Clarification June 15, 2014:   Struck through the last line, and added a replacement line that communicates the implications of Jon Greenberg's calculation more clearly.