How to bulletproof your data story

By Shane Sanderson

When the Palm Beach Post obtained a spreadsheet made by a clerical worker at the local medical examiner’s office, reporters had to verify it.

The office worker had noticed an escalation in the number of overdose deaths and she began a project recording the details. The resulting spreadsheet had something like 100 columns, full of dirty and perhaps inaccurate data.

Journalists ended up requesting and receiving cover pages from the medical examiner’s original reports, police reports, and reports from the Florida Department of Law Enforcement. Then Joel Engelhardt’s team called family members of the 216 people that died from heroin overdoses in Palm Beach County.

Engelhardt shared his verification story as part of a panel with Jennifer Forsyth of the Wall Street Journal on bulletproofing data stories.

Forsyth talked about her “nightmare,” in which she imagined she would have to take the stand in a lawsuit against her news organization for a story she had not properly verified. To avoid that scenario, she asks a series of questions of her data reporters:

• Do we have the universe or just a subset? How did you select your subset? Is it representative?
• If you have a percentage in the story? How many people is that calculation based on?
• If data came from somewhere other than the original source, how do you know the data is up to date? Can we check against the original source?
• Was the data self-reported? Was there rigorous vetting of self-reported data? How do you know?
• When matching datasets on the basis of names without other corroborating information, how did you verify those are the people you think they are? Did you verify every name or just spot check?

Forsyth also spelled out best practices for editors:

• Sit down with the reporte and have her show you the spreadsheets and calculations used in the story.
• Ask the reporter to send in a detailed explanation of her data reporting methodology early in the process.
• Ask the reporter for the record layout so you know what data was chosen and what was left out.
• Resist the urge to write around anything that is inconvenient or not easily explained.
• Ask another IRE member for help.