Your average CAR geeks - especially the old timers - follow a predictable route in tools they use for data analysis and sharing.

You start with Microsoft Excel spreadsheets, and everything’s going fine. But then someone tells you about relational databases, and suddenly you notice all of the things you can’t easily do in Excel.

Step up to Microsoft Access database manager and pretty soon you’re joining tables right and left, slipping terms like "Group By" and "normalization" into conversations and generally feeling pretty good about yourself.

But at some point, someone in your newsroom looks over your shoulder ...

Read more ...

NOTE: This tutorial assumes a basic familiarity with the Python language and interpreter, an interactive environment for running code. It also requires Python 2.6 or later.

As a database editor, my daily routine often involves pulling data from spreadsheets and databases and processing it for a variety of other tasks: automated email alerts, visualizations, mashups with data from other sources.

The moment data leaves the confines of Microsoft Excel spreadsheets or a relational database manager, you lose all those handy column headers that help sort, filter and otherwise make sense of information. This recipe shows how, using Python, you ...

Read more ...

In this guide, we're going to walk through the process of scraping and cleaning data from the web in real time, using only Google Spreadsheets. As an example, I'll be using Columbia 911, a site I put together for this purpose.

Google Spreadsheets are the ultimate weapon when it comes to real-time data and Web-based mashups. There's a slight learning curve, but once you get over it you'll find Google Spreadsheets can do just about anything Microsoft Excel spreadsheets can. Even better, they're hosted in the cloud, which means they're almost always available online ...

Read more ...

I am constantly getting data from all kinds of public agencies that provide the most important field – the date of birth – in different formats. An example is city employee data. I request the name, DOB, salary and job title from San Antonio and other nearby cities to use in determining whether any of those workers have a criminal record.

Most of the time, the agencies provide the birth date like this in text format: mm/dd/yyyy. I prefer to work with the date like this in text format: yyyymmdd. When I want to find out how many employees were ...

Read more ...

For journalists hoping to analyze geographic data, the cost of geographic information system (GIS) software can be prohibitive. Fortunately, there are open-source, free solutions available for cost-cutting journalists who want to do spatial analysis.

Pursuing an open-source option is easier said than done. Often, open-source software can be difficult for the lay journalist to install and even begin to understand.

PostGIS for PostgreSQL database manager offers a solution that is free, robust and easy to use — assuming you know what you're doing.

Granted, when I received the assignment to install PostGIS for PostgreSQL on my Mac to handle spatial ...

Read more ...

Linear regression is a great tool when your outcome variable is test scores or loan amounts or another continuous variable. But sometimes, your output is a Yes or a No. That type of outcome is known as dichotomous. You still can do something similar to linear regression because some super smart stats dude awhile back came up with a way to mimic linear regression with a dichotomous outcome variable. To do logistic regression SPSS, you need to have the “regression models” add-on program. You also have to understand your data and do a little prep work on it. As with ... Read more ...

The other day, I was talking to one of my colleagues about how I rarely use ArcView anymore. Since becoming a full-fledged Linux jockey, I've found so many tools that process GIS data better than Arc ever has. Sure, it still provides a pretty graphical interface, which definitely comes in handy, but most of the time I don't need it. The green and black of the Linux terminal window line suits me just fine. Before you dismiss my flagrant geekdom, let me make this case: Whenever a question comes up on NICAR-L about how to accomplish some uncommon ... Read more ...

One of the biggest hurdles we all face for trying new software or utilities is the lack of a sandbox, a machine we can just use when we want to without having to worry if something goes wrong. This is particularly true for new open-source technologies, like the fast-growing field of open source GIS software. Sure, it would be great to try out OpenLayers or other mapping utilities, but it's not like we can just turn our main computer into a development box overnight. Luckily, this problem has been worked out for us. In the past week, I've ... Read more ...

So, a tipster has just shown you the document you need to nail a story. Problem is, she can't give it to you, there's no copy machine in the diner where you met and you need to show it to your editor before the story runs. What to do? What to do? Enter Qipit, a free service that turns pictures from your cell phone into searchable PDFs. Using Qipit is as simple as taking a picture of a document, white board or even handwritten notes with your QIPIT-compatible phone and sending the image in text message or e-mail ... Read more ...

When you have detailed information about small areas, an interactive, thematic Web map can make all the difference, such as this one showing presidential election results by precinct. For many news organization facing cutbacks, the financial investment can be daunting. Commercial geographic information system products are expensive, as are the products that can translate them to the Web in an interactive form. Fortunately, recent improvements in open-source mapping have put these innovations within reach of anyone with the patience to learn how to use them. In this tutorial, you’ll make an interactive map of the New York region that ...

Read more ...