Don't let the robots win

  • Event: 2016 CAR Conference
  • Speakers: Liam Andrew of The Texas Tribune; Daniel Craigmile of The Texas Tribune
  • Date/Time: Sunday, Mar. 13 at 10:15am
  • Location: Denver III & IV
  • Audio file: Only members can listen to conference audio

Bots, crawlers, scripts, and scrapers: at NICAR you can learn to build them, but can you defend your site from them? Whether they're built by data journalists, search engines, web archivists or malicious troublemakers, bots are substantial users of news websites. At The Texas Tribune, bots account for half of our site's traffic, and they're responsible for our search presence and our archival legacy as well as site attacks and performance problems. In this session, we'll dive into the world of bot users, and share some tips for identifying and managing these crawlers and scrapers, helping the "good" bots do their work, and keeping the "bad" ones from wreaking havoc.

Speaker Bios

  • Liam Andrew is a developer at The Texas Tribune, where he specializes in building editorial tools and API integrations. He recently completed MIT's Comparative Media Studies masters program, researching news archives and structured journalism, and designing and developing tools for digital annotation, collaborative publishing, and news and event discovery. He was a researcher at MIT HyperStudio and a Google Journalism Fellow with the Nieman Lab. @mailbackwards

  • Daniel Craigmile does technical things at The Texas Tribune. @x110dc

Related Tipsheets

  • Don't let the robots win
    Bots, crawlers, scripts, and scrapers: at NICAR you can learn to build them, but can you defend your site from them? Whether they're built by data journalists, search engines, web archivists or malicious troublemakers, bots are substantial users of news websites. At The Texas Tribune, bots account for half of our site's traffic, and they're responsible for our search presence and our archival legacy as well as site attacks and performance problems. In this slideshow, we'll dive into the world of bot users, and share some tips for identifying and managing these crawlers and scrapers, helping the "good" bots do their work, and keeping the "bad" ones from wreaking havoc.