By Anna Boiko-Weyrauch
@AnnaBoikoW

Syntax error. What does this bit of code do? Syntax error. Let’s go back to the source. Syntax error. Maybe try this?

After two hours of educated guesses, trial, error and some friendly help, Pam Dempsey, of cu-citizenaccess.org, and I had finally scraped our first bit of text: the word “2011” from a page of Illinois nursing home inspection records.

The ScraperWiki session on Thursday night was aimed at “liberating” a number of data sets from their online prisons by working late into the night scraping real websites.

The session started with a two-hour ...

Read more ...

Your average CAR geeks - especially the old timers - follow a predictable route in tools they use for data analysis and sharing.

You start with Microsoft Excel spreadsheets, and everything’s going fine. But then someone tells you about relational databases, and suddenly you notice all of the things you can’t easily do in Excel.

Step up to Microsoft Access database manager and pretty soon you’re joining tables right and left, slipping terms like "Group By" and "normalization" into conversations and generally feeling pretty good about yourself.

But at some point, someone in your newsroom looks over your shoulder ...

Read more ...

Data entry hell.

We’ve all been there. It’s the place where a team of 17 enters information off of handwritten expense forms into a database. It’s the no-man’s land where your summer interns type in the results of three months’ worth of records requests. It’s you combining summer camp listings from no fewer than 12 spreadsheets, each with its own adorable little quirks and styles.

Often, newsrooms tackling these kinds of monster projects turn to the least-common denominator method in the name of convenience. Generally, that means using Microsoft Excel spreadsheets or an Access database ...

Read more ...

For journalists hoping to analyze geographic data, the cost of geographic information system (GIS) software can be prohibitive. Fortunately, there are open-source, free solutions available for cost-cutting journalists who want to do spatial analysis.

Pursuing an open-source option is easier said than done. Often, open-source software can be difficult for the lay journalist to install and even begin to understand.

PostGIS for PostgreSQL database manager offers a solution that is free, robust and easy to use — assuming you know what you're doing.

Granted, when I received the assignment to install PostGIS for PostgreSQL on my Mac to handle spatial ...

Read more ...

Gary Sherman's Desktop GIS: Mapping the Planet with Open Source Tools

When journalists are looking for software, they usually greet the words "open source" in one of two ways: with confusion, because open source software is still a daunting mystery; or with delight, because the software is available for free.

Open source software is simply software that is available at no cost and has its source code available to the public. A network of users and developers constantly enhances and expands the program.

Journalists doing CAR have been using robust open-source tools for editing documents, analyzing spreadsheet data and ...

Read more ...

By now you may have read that EveryBlock, a Knight Foundation-funded project, has released its source code to the public (here's a browsable version). Getting a chance to look under the hood is a great opportunity to see how other folks tackle some of the tasks we all face, or are likely to. The first thing to note is that the code has the GPL license, which means that if you incorporate any of it into an application you're building and then release that code, it will need to be under the terms of the GPL as well ... Read more ...

I had the privilege of speaking on a panel with Sarah Cohen and Steve Doig last week in Baltimore about the future of computer-assisted reporting. Whoever thought I even belonged in the same room as those two gave me way more credit than I deserved. But in preparing for that panel, I got to thinking: What skills and software tools are we going to be using in 10 years? What skills should we start learning now if we want to be prepared for the future? Or better yet: What types of problems in newsgathering and investigations could technology best help ... Read more ...

As news outlets struggle to make bottom line, journalists might look for new ways to save some cash. Fortunately, journalists doing CAR can look toward no-cost open-source database managers as powerful alternatives to commercial software. Open-source database applications like MySQL and PostgreSQL are more resilient than commercial software applications, such as Microsoft Access, and offer more powerful capabilities for analyzing and synthesizing data. Glen McGregor, database manager for the Ottawa Citizen, used to rely on Access to query data he received from the federal government. When McGregor made a request to the federal government to release data revealing where guns ...

Read more ...

Outside hundreds of schools across the country, children are exposed to air that appears to be rife with chemicals that can exacerbate asthma or cause cancer. The reporting that led us to that astounding conclusion began with a relatively straightforward question: What's in the air outside the nation's schools? To find the answer, we turned to a variety of state and federal databases that pinpoint schools, detail the chemicals released from industrial facilities and estimate the potential severity of toxic air pollution across the country. Many newspapers have dealt extensively with the government's most basic pollution information ...

Read more ...

The other day, I was talking to one of my colleagues about how I rarely use ArcView anymore. Since becoming a full-fledged Linux jockey, I've found so many tools that process GIS data better than Arc ever has. Sure, it still provides a pretty graphical interface, which definitely comes in handy, but most of the time I don't need it. The green and black of the Linux terminal window line suits me just fine. Before you dismiss my flagrant geekdom, let me make this case: Whenever a question comes up on NICAR-L about how to accomplish some uncommon ... Read more ...