The methodologies of fact-checking

• FEDERAL
• STATE
• LOCAL
• BY DATE
• POLICY
  • Agricultural policy
  • Budget policy
  • Campaign finance policy
  • Criminal justice policy
  • Defense and foreign policy
  • Drug policy
  • Education policy
  • Energy policy
  • Environmental policy
  • Gun policy
  • Government finance policy
  • Government regulation policy
  • Healthcare policy
  • Immigration policy
  • Tax policy
• ABOUT FACT CHECK

---

Fact Check was Ballotpedia's fact-checking project that ran from October 2015 through October 2018. These pages have been archived.

---

Fact checkers use a range of methodologies that shape their approach to fact-checking. These methodologies govern how they select topics, conduct research, structure articles, evaluate claims and—most importantly—render verdicts.

Below, we break down the methodologies of three major fact-checking organizations in the United States: PolitiFact, FactCheck.org and The Washington Post Fact Checker. We focus on the following areas:

1. Selection process: the processes by which fact checkers choose what claims to evaluate.
2. Research methods: the basic techniques and types of sources that fact checkers most commonly use when conducting research on claims, as well as the official rules and editorial policies that govern their approaches.
3. Claim evaluations: the systems and processes by which fact checkers establish the veracity of a claim.

PolitiFact

See also: PolitiFact

PolitiFact began in 2007 as a project of the St. Petersburg Times (now known as the Tampa Bay Times), calling itself "an independent, nonpartisan news organization." Bill Adair, the St. Petersburg Times' Washington bureau chief, and news technologist Matthew Waite founded PolitiFact with backing from the newspaper.

PolitiFact explains some of the methodology used on its website in an "About PolitiFact" section and in an article ("Principles of Politifact and the Truth-O-Meter") written by PolitiFact creator Bill Adair on February 21, 2015. Other sources of information for PolitiFact methodology include interviews with staff and outside observers.

Selection process

PolitiFact fact checks a wide range of political actors and groups, including both elected and non-elected government officials, political candidates, media pundits, celebrities, and special interest groups. They also examine claims made in social media circles in the form of memes and viral images.

But how do they go about selecting claims to fact check?

According to his articles on PolitiFact's principles, Adair says their selection process begins with a preliminary sweep through news stories, political ads and speeches, campaign websites, social media, and press releases. They also accept suggestions from readers.

Adair says that PolitiFact asks these critical questions when determining what to fact check:

“

Is the statement rooted in a fact that is verifiable? We don’t check opinions, and we recognize that in the world of speechmaking and political rhetoric, there is license for hyperbole. Is the statement leaving a particular impression that may be misleading? Is the statement significant? We avoid minor 'gotchas' on claims that obviously represent a slip of the tongue. Is the statement likely to be passed on and repeated by others? Would a typical person hear or read the statement and wonder: Is that true?

”

Another key factor in PolitiFact's selection process is newsworthiness. Adair wrote that the site selects "the most newsworthy and significant" claims. He is reported to have told journalism students in 2011, "We're guided by news judgement. And we are all journalists, we're not social scientists."

Adair has cited curiosity as an additional factor that shapes PolitiFact's selection process. In an August 2009 interview for C-SPAN's "Washington Journal," he said, "We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact check it."

Research methods

PolitiFact's research generally begins with contacting the source of a claim, according to Adair and researcher Lucas Graves. If an elected official, for example, makes a claim that attracts PolitiFact's attention, they reach out to that individual's office for clarification, data or a source to back up that claim. Graves noted that this step is required of PolitiFact writers.

With or without data from the original source, PolitiFact then turns to news articles, free and subscription-based sources on the Internet, and on-record interviews with reporters and experts on the subject. On-record interviews assist writers and researchers with the interpretation of data. PolitiFact prohibits the use of off-record sources.

According to Graves, PolitiFact writers are encouraged to seek out nonpartisan data sources whenever possible. Such sources often include government agencies like the Bureau of Labor Statistics or the Congressional Budget Office.

PolitiFact publishes a list of its sources along with each article.

Claim evaluation

PolitiFact's uses a group approach for settling the veracity of a claim.

The first step involves the lead writer submitting an article with a recommended rating to a panel of at least three editors, according to Adair's "Principles of Politifact" article as well as researchers and reporters who have observed the PolitiFact evaluation process.

PolitiFact's rating system is the "Truth-o-Meter." The meter has six ratings:

PolitiFact's Truth-o-Meter

“

TRUE – The statement is accurate and there’s nothing significant missing. MOSTLY TRUE – The statement is accurate but needs clarification or additional information. HALF TRUE – The statement is partially accurate but leaves out important details or takes things out of context. MOSTLY FALSE – The statement contains an element of truth but ignores critical facts that would give a different impression. FALSE – The statement is not accurate. PANTS ON FIRE – The statement is not accurate and makes a ridiculous claim.

”

The panel of editors evaluates the article and the author's recommended rating. The panel then discusses whether PolitiFact should follow the author's recommendation or assign a new rating. To make this decision, Adair writes, the panel of editors relies on five principles:

“

Words matter -- We pay close attention to the specific wording of a claim. Is it a precise statement? Does it contain mitigating words or phrases? Context matters -- We examine the claim in the full context, the comments made before and after it, the question that prompted it, and the point the person was trying to make. Burden of proof -- People who make factual claims are accountable for their words and should be able to provide evidence to back them up. We will try to verify their statements, but we believe the burden of proof is on the person making the statement. Statements can be right and wrong -- We sometimes rate compound statements that contain two or more factual assertions. In these cases, we rate the overall accuracy after looking at the individual pieces. Timing – Our rulings are based on when a statement was made and on the information available at that time.

”

Graves, who sat in on 25 editor evaluation meetings, said that "two of three panelists must agree on the final ruling, though unanimity is strongly preferred and usually achieved." He also said that meetings often last 10 to 15 minutes, though disagreements among the editors or between the editors and the author occasionally extend meetings to multiple hours and sometimes require follow-up meetings. In one instance, Graves reported, an "indecisive panelist had to be replaced" before the panel could make a ruling.

FactCheck.org

See also: FactCheck.org

FactCheck.org was founded in 2003 by former AP, Wall Street Journal and CNN reporter Brooks Jackson and academic Kathleen Hall Jamieson with backing from the Annenberg Public Policy Center of the University of Pennsylvania.

FactCheck.org provides explanations of its methodology in a brief mission statement on its website and in two articles written by former Director Brooks Jackson. Additional information on its methodology, policies and approach to fact-checking is derived from outside sources.

Selection process

FactCheck.org targets "major U.S. political players in the form of TV ads, debates, speeches, interviews and news releases," according to a page titled "Our Mission" on its website. Like PolitiFact, FactCheck.org also fact checks social media claims and chain emails. Unlike PolitiFact, though, it tends to avoid fact-checking media figures and pundits.

Lucas Graves, who sat in on a FactCheck.org training session for his dissertation on fact-checking, notes that the organization emphasizes verifiable facts over opinions in its selection process. He says they typically pursue only claims that are false. When it becomes apparent that a claim is true, the research often ends there and a new claim is picked up. Moreover, FactCheck.org seeks out claims with "national significance" that most readers would find interesting.

Research methods

FactCheck.org, like PolitiFact and The Washington Post Fact Checker, contacts the individual or organization responsible for a claim to request backup data and/or original sources. Writers and researchers also often try to trace claims back to original sources by tracking news stories that mention a specific claim or statistic.

Graves notes that FackCheck.org writers and researchers are taught to rely heavily on federal and state government agencies for raw data. A list of resources compiled by FactCheck.org Director Eugene Kiely shows that they make use of nonprofit groups and think tanks as well, including the Tax Policy Center, the Tax Foundation and the Center for Effective Government.

The organization regularly makes use of experts to help interpret data. According to Graves, though, they do this far less than PolitiFact. FactCheck.org avoids using anonymous sources. Former director, now director emeritus, Brooks Jackson has been quoted as saying, "[W]e don’t cite anonymous sources as proof of anything factual. Why would anyone believe it if we did ... We think of our pieces as meeting the high standards of academic scholarship."

FactCheck.org includes a list of sources at the end of each article.

Claim evaluation

FactCheck.org does not use an official rating system. It is unique among the three major fact-checking organizations in the United States for this reason. Jackson addressed the issue in a December 2012 article titled "Firefighters, Fact-Checking and American Journalism." He called rating systems "inflexible." He said:

“

Rating statements with devices such as 'truth-o-meters' or 'Pinocchios' are popular with readers, and successful attention-grabbers. But such ratings are by their nature subjective — the difference between one or two 'Pinocchios' is a matter of personal judgment, and debatable. Some statements are clearly true, and some provably false, but there’s no agreed method for determining the precise degree of mendacity in any statement that falls somewhere in between.

”

Jackson echoed this in an April 2015 article, saying, "We consider [rating systems] inherently subjective, and find that many political claims don't fit neatly into inflexible categories."

While FactCheck.org writers usually assert whether they found a statement to be true, false or somewhere in between, they often qualify the ruling with additional context. Jackson cited an example from PolitiFact and compared it to how FactCheck.org would have handled the claim:

“

A senator who said a 'majority' of Americans are conservative was rated 'mostly true' (and later 'half true') even though the statement was false. The story cited a poll showing only 40 percent of Americans rated themselves conservative. That’s more than said they were moderate (35 percent) or liberal (21 percent) but still far from a majority. The senator had a point, but stated it incorrectly, thereby exaggerating. A simple 'truth-o-meter' had no suitable category for that. Our approach would have been to say that it was false. But we would also note that the senator would have been correct to say Americans are more likely to call themselves conservative than moderate, or liberal, when given those three choices.

”

In an American Press Institute "Fact-Checking Project" study, this kind of approach to claim evaluation is described as "a nuanced contextual analysis of the contested claim." The study added, "These fact checks may refute egregious claims in clear, decisive language, announcing in the headline or the first sentence that a speaker has distorted the truth. But they stop short of assessing statements in any systematic fashion that would allow different claims or speakers to be compared." FactCheck.org's method of claim evaluation is, therefore, fluid and encourages readers to examine claims and those who make them on a case-by-case basis.

The Washington Post Fact Checker

See also The Washington Post Fact Checker

The Washington Post Fact Checker started in 2007 as a temporary project aimed at fact-checking the 2008 presidential campaign. The Washington Post made the Fact Checker a permanent feature in 2011 under reporter Glenn Kessler.

Glenn Kessler provided a brief overview of The Washington Post Fact Checker's mission, methodology, principles and scoring system in an "About the Fact Checker" article published in September 2013. Outside sources and interviews with Kessler provide additional information.

Selection process

The purpose of The Washington Post's Fact Checker column, Kessler wrote, is "to 'truth squad' the statements of political figures regarding issues of great importance, be they national, international or local." Political figures, in this case, include elected and non-elected government officials and "political candidates, interest groups, and the media."

The column places a heavy emphasis on the role of readers in selecting claims to fact check. Kessler said, "It's a big world out there, and so we will rely on readers to ask questions and point out statements that need to be checked." He added, "the success of this project depends, to a great extent, on the involvement of you—the reader. We will rely on our readers to send us suggestions on topics to fact check and tips on erroneous claims."

He also included a series of principles for the Post's Fact Checker. The principles that shed light on his claim-selection process are included below:

“

This is a fact-checking operation, not an opinion-checking operation. We are interested only in verifiable facts, though on occasion we may examine the roots of political rhetoric. We will focus our attention and resources on the issues that are most important to voters. We cannot nitpick every detail of every speech. We will strive to be dispassionate and non-partisan, drawing attention to inaccurate statements on both left and right.

”

Research methods

Similar to its counterparts, PolitiFact and FactCheck.org, the Post's Fact Checker reaches out to the individual or organization responsible for a claim and uses raw data and original sources to examine it. Kessler said that Google searches can be effective tools for investigation, telling Lucas Graves in an interview in 2012, "When I encounter a weird [or] strange fact, one of the first things I do is google the figures, which is an amazingly efficient way to figure out where it comes from."

Unlike his counterparts, however, Kessler said he prefers not to consult subject experts, even for assistance with interpreting complex data. "I'm the kind of reporter who is reasonably confident in his judgments. I like to speak with my voice, because I have actually covered just about everything in Washington... You can see my bio, you can see what I’ve covered. I bring a unique perspective of having listened to bull***t in Washington for 30 years," he told Graves in an interview.

In contrast to PolitiFact and FactCheck.org, the Post's Fact Checker will, in some cases, cite anonymous sources.

Claim evaluation

The Post's Fact Checker's approach to evaluating the veracity of a claim, Kessler says, centers on a "'reasonable man' standard for reaching conclusions. We do not demand 100 percent proof."

Like PolitiFact, the Post's Fact Checker uses a rating system. It is based on Pinocchio, the title character of the children's story, whose nose grew longer with each lie he told. For false claims, it assigns up to four Pinocchios to a statement depending on how untrue the claim is determined to be. Kessler provides a guide to the Pinocchio system in his "About the Fact Checker" article. It can be seen below.

A Pinocchio head

• One Pinocchio: "Some shading of the facts. Selective telling of the truth. Some omissions and exaggerations, but no outright falsehoods. (You could view this as 'mostly true.')"

• Two Pinocchios: "Significant omissions and/or exaggerations. Some factual error may be involved but not necessarily. A politician can create a false, misleading impression by playing with words and using legalistic language that means little to ordinary people."

• Three Pinocchios: "Significant factual error and/or obvious contradictions."

• Four Pinocchios: "Whoppers."

An additional factor that the Post's Fact Checker uses for assigning Pinocchios is whether or not a claim has been previously debunked. Kessler writes, "Repeated misstatements of previously debunked statistics can, over time, result in higher Pinocchio ratings for a particular claim. In other words, we may hold a politician to a higher standard if he or she already has been put on notice that a certain 'fact' is dubious."

For claims that turn out to be true, the Post's Fact Checker awards a Geppetto check mark, a reference to Pinocchio's father/creator who had a reputation for telling the truth. They also use an upside down Pinocchio head for "flip-flops" and a scales of justice symbol for claims that are too difficult to verify or require more time and/or data.

See also

• What is fact-checking?
• Fact-checking organizations
• International fact-checking sites
• Academic studies of political fact-checking
• PolitiFact
• FactCheck.org
• The Washington Post Fact Checker

External links

• PolitiFact.com, "About PolitiFact"
• PolitiFact.com, "Principles of PolitiFact and the Truth-O-Meter," February 21, 2011
• FactCheck.org, "Our Mission"
• FactCheck.org, "Firefighters, Fact-Checking and American Journalism," December 21, 2012
• The Washington Post Fact Checker, "About the Fact Checker," September 11, 2013