College accreditation might sound like a wonky subject – the kind of thing that only interests policy makers and people at think tanks. But accreditors have tremendous power. Colleges have to be accredited in order to get federal student aid. This system is the main thing standing between colleges and billions of dollars – $134 billion to be exact.

My colleague Doug Belkin and I got interested in exploring this system after wondering how so many schools with really low graduation rates and high loan default rates get accredited. From covering higher education, we knew that the Department of Education entrusts accreditation to non-profit groups that review colleges. But we didn’t really know much about the inner workings of these groups.

I had an inkling from previous reporting that accreditation had a lot more to do with making sure a college was financially solvent than its quality. So I proposed a simple data question: how do accredited colleges stack up against schools that were denied? If colleges still in the system were much lower performing than those that had been stripped of their accreditation, then we had our story. Were the gatekeepers of billions of dollars focused on weeding out low performers, or on something else entirely?

This was trickier than I imagined. You have to know a lot about Department of Education data called IPEDS (the Integrated Postsecondary Education Data System), which Holly Hacker of the Dallas Morning News addressed in an earlier Uplink post. IPEDS has a nifty website where you can pull data extracts.

I’d worked with this system before, but this time I ran into a problem: the masters of IPEDS removed closed schools from the search function, and most schools close when they lose accreditation. Fortunately for me, the original data was still in IPEDS raw data files.

It’s possible to look in this data in SAS or SPSS statistics programs, but I wrote a Python script to download all of the raw data files. There are dozens of tables, and the Department of Education publishes a separate file each year for each table. These files aren’t in a standard format – sometimes they add new columns or change the position of existing ones. So my script looked for the same column names across multiple files and output one big file for each table. I imported these into Microsoft SQL Server after using my script to figure out column sizes and types.

I’d only recommend these raw data files to an expert IPEDS user; the tool that allows you to create custom exports will give most people what they need. I had to read a lot of code manuals to figure out how to use these raw files.

I exchanged a lot of emails with the staffer at the Department of Education who runs the database to double-check my math. Talking to the actual people who maintain your data – or at least some other expert in it – is a must. They may act like you are driving them crazy, but it’s the only way to make sure you’re doing your job right. With data, to paraphrase Donald Rumsfeld, there are known unknowns, and there are unknown unknowns.

There’s no field that says “graduation rate.” I had to add a lot of underlying variables together to get this number. Once I did that, I had the graduation rates of all schools that lost accreditation. I looked at the rates in the year before they lost accreditation, but as a gut check, I made sure there was nothing anomalous about that data by looking across several years.

We did a similar analysis for default rates. The Department of Education now publishes default rate data based on a three-year period, but it has data back to the 1990s on students who defaulted within two years of leaving school. Fortunately, these were in Excel files so they were easier to import into SQL server.

The result: Nearly 350 out of more than 1,500 accredited four-year schools had a lower graduation rate or higher student loan default rate than the average of the schools that lost accreditation.

The average graduation rate of schools that lost accreditation was 35 percent, but we found some accredited schools that had graduation rates in the teens and single digits. Similarly, the average default rate of a school that lost accreditation was 9 percent. Some schools have default rates two or three times as high.

We were also interested in seeing how much in federal student loans and grants these schools receive, given that accreditation is required to receive federal student aid. Were schools getting billions of dollars because they skated through their accreditation reviews?

For this, we used data from the Federal Student Aid data center that shows student loan disbursements by quarter. We imported these Excel files into SQL Server and found that last year, the government gave $16 billion in loans and grants to students at colleges graduating less than a third of people who enroll.

All the while pursuing data analysis, we filed over 100 freedom of information requests to public colleges, seeking their latest accreditation reviews. We read through the documents we received and found cases where graduation rates and default rates were never mentioned at some of the lowest-performing schools. In fact, some of those schools received glowing praise.

We also found through our reporting that accreditors have only removed 18 four-year colleges from membership since 2000. The representatives of those groups argued that it was their job to help schools get better, not to police schools and kick out the worst ones.

Accreditors are serious about their jobs. They put a lot of work into reviewing schools. But we ultimately found that there’s a disconnect between what they see as their mission and the watchdog role the government has tasked them with. As a result, the watchdogs over higher education rarely bite.

 

Andrea Fuller is an investigative report at The Wall Street Journal who specializes in data analysis. Contact her at andrea.fuller@wsj.com.