When the Great Recession reached its peak at the end of 2009, a lot of what you heard and read from news organizations about the state of the economy felt depressingly similar.

Long lines at unemployment offices. Big crowds at job fairs. And everywhere, mounting horror stories from families struggling harder than ever before to make ends meet.

Amid this pervasive gloom, some supposed bright spots popped up regularly in our inboxes, courtesy of the North Carolina governor’s press office.

New jobs, as few as six and as high as a 1,014, were on their way to the state’s ailing counties as companies sought to relocate or expand. These jobs and new investments came with the promise of incentives, subsidized by taxpayer money.

Collectively, we reporters and producers relayed these announcements without much skepticism, evidence that the state’s sputtering economy was showing at least small signs of turning over.

Screenshots from WRAL’s series

But the more releases I received, the more I began to wonder: How often do these job promises pay off?

In early 2014, almost five years after that question wormed its way into my brain, I finally got the chance to answer it as an investigative reporter for WRAL News’ state politics team.

By then, the topic of corporate subsidies had already been heavily explored. A 2012 New York Times investigation put the price tag of these state and local incentives at $80 billion annually, a figure they noted was probably an underestimate. Atlanta Journal Constitution reporters found about half the companies didn’t deliver the numbers touted by Georgia leaders. And in Indiana, the investigative team at WTHR found serious gaps in the way state leaders were reporting job creation numbers under a new transparency law.

I wasn’t exactly sure how to attempt something similar, but I knew I needed data.

So I started with those press releases. Around the end of 2013, I went to the state archives with a request for every press release from the administration of Democratic Gov. Bev Perdue.

We were already a year into the tenure of Republican Gov. Pat McCrory. But job announcements – and the incentive cash that comes with them – often mean expansion or relocation efforts that can take years. My hunch was Perdue’s tenure from 2009 to 2012 was an ideal period to examine.

State archivists in January delivered a PDF collection of every press release, as well as a rough spreadsheet containing valuable data like titles, dates and text of the releases.

On and off over the next several months, I went through the collection to see what I could do with it. I pitched the rows mentioning anything other than jobs and pretty soon realized that these announcements followed a very consistent format.

In almost every case, the text contained the number of jobs projected, average wage, the value of the incentive and other data like county and city where the project would be located. It wasn’t consistent enough to extract with code, so I manually went through the text field for all of the more than 200 announcements, placing the info into new columns in the existing spreadsheet to provide more structure to the data.

In retrospect, this was an odd move. I was juggling a number of other projects at the time, and I didn’t check what data the N.C. Department of Commerce actually tracked. But I was hesitant to start with state data without knowing how reliable it might be.

For perspective on that, I called the nonprofit Good Jobs First in May. The think tank has been hammering away at the issue of subsidies for years, and researchers there were able to give me a good sense of what data to look for and what pitfalls to avoid.

With their guidance, I found the state Department of Commerce has a ton of great information that notes what it’s doled out and clawed back in cases where companies don’t deliver. But the most valuable data I found specifies how many jobs are actually hired based on verified tax filings.

There was just one problem: the data posted on the Web in PDF form contained no company names. I soon learned that by law, much of the information collected on company job performance and reported to commerce officials and legislators is stripped away before it’s posted online every year.

It took a solid month to get ahold of the unedited spreadsheet that the PDF was based on, but it was a gold mine.

A spreadsheet of raw state data used for the job performance piece

By combining the actual job data with my existing data on what the governor’s press office promised using Microsoft Access, I was able to a calculate a few major statistics. Two were self-explanatory: the percentage of jobs created and the percentage of money the state had so far paid out.

But the third was crucial to putting the former two in context.

Subsidy programs all have different “ramp-up” periods, which give companies time to meet their goals as they build new facilities and hire workers. To gauge whether these projects were “on track,” at least by the measure of living up to their promises, I used state data to calculate how much of each grant period had elapsed.

The investigation showed about 38 percent of the promised jobs have so far come to fruition, although North Carolina had only paid out 12 percent of the money. Collectively, the projects were 75 percent of the way through their grant cycles.

Those statistics drove one of the four pieces in the series my colleague Mark Binker and I wrote on the role of job incentives in North Carolina’s economic development. The data also powered an application I hacked together using a few pre-built Javascript libraries – Leaflet for mapping and Footables for responsive tables – which allowed readers to explore the data on the state or local level. With the help of Alex Phillips on our WRAL.com tech team, we were able to implement the app right into our content management system.

We also turned the more print-centric, online work into television pieces with a promo, multiple reporter Q&As and a package fronted by Capitol Bureau Chief Laura Leslie, who broke the complex story down for viewers in about three minutes.

To show our work, we published a detailed methodology and uploaded all the data on Github, two things we planned to do from the start.

The great thing about this project is that it’s infinitely reusable. We no longer have to report on job announcements without the context of how often they fall short. With a click or a tap, we can look up previously incentivized companies that announce massive layoffs or receive additional subsidies without living up to past commitments.

And with a defined roadmap for how to measure progress, we can update our findings as new data is published – and hold future governors accountable for their promises.

Tyler Dukes (@mtdukes) is an investigative reporter on the state politics team at WRAL News in Raleigh, N.C., where he specializes in data and public records. Prior to joining WRAL, he worked as managing editor for the Duke University Reporters’ Lab, a research project aimed at reducing the costs of investigative reporting. He’s taught class at the UNC-Chapel Hill’s journalism school, freelanced for The News & Observer and trained reporters as an adviser to a daily student newspaper. Contact him at [email protected].