And we’re looking for a few good hackers. Please spread the word, and email brian@hackerjournalist.net.

The city dropped 10 years of incredibly detailed crime information a
few weeks ago, and we’ve barely touched it. The state of Illinois just
released an 9500-column-wide data set on school performance. And there may just be a few important elections peeking over the horizon.

The Chicago Tribune needs you. Your city needs you.

Join us.

Photo by Chrys Wu

Today at the MIT/Knight Civic Media Conference, I led an unconference session called “Our sci-fi future, news in 20 years”. Chrys Wu was kind enough to transcribe the concepts we discussed and help me annotate the list with related works of science fiction. Here’s what we wrote down:

Influencing factors are in italics, related books and films are in bold:

All your data in the cloud

sukey.org

Instantaneous backstory (I want the machine to know what I know and only give me what’s news to me)
Diamond Age

Worldview mapping

Conversion tracking (like in advertising) for information, track the impact of the reporting

Virtual connections w/ physical stimuli

Implants for sensory input
Accellerando
Altered Carbon

Heads up displays
Mona Lisa Overdrive

Real-time maps of information spread

Millions of little flying cameras, ubiquitous surveillance
Counting Heads

Infinite bandwidth, infinite processing

Machine translation/transcription (babelfish)
Hitchhikers Guide

Computer Q&A

More ambient experiences (like the radio, not the radio)
Air

Individually relevant metrics (Mint, Bedpost)

Gargoyles (permanently plugged in)
Snow Crash

The singularity (when human and machine become indistinguishable)
The Way of All Flesh
House of Suns

Info valet / personal information assistant
Counting Heads

The map of information consumption

Augmented Reality
Rainbows End

Pervasive advertising
Jennifer Government

Attention markets (trade your attention, certain people’s time worth more than others)
Crystal Express
Little Brother

Embodied news narratives

Authentication/authority systems
A Fire Upon the Deep

Other reads, unassociated with a particular topic listed above:
Maul, Tricia Sullivan
Summer Wars
Daemon
Ghost in the Shell
Iron Sunrise
Marq’ssnan Cycle
The Information

Photo by Waldo Jaquith

Brother Cavil: In all your travels, have you ever seen a star go supernova?

Ellen Tigh: No.

Brother Cavil: No? Well, I have. I saw a star explode and send out the building blocks of the Universe. Other stars, other planets and eventually other life. A supernova! Creation itself! I was there. I wanted to see it and be part of the moment. And you know how I perceived one of the most glorious events in the universe? With these ridiculous gelatinous orbs in my skull! With eyes designed to perceive only a tiny fraction of the EM spectrum. With ears designed only to hear vibrations in the air.

Ellen Tigh: The five of us designed you to be as human as possible.

Brother Cavil: I don’t want to be human! I want to see gamma rays! I want to hear X-rays! And I want to – I want to smell dark matter! Do you see the absurdity of what I am? I can’t even express these things properly because I have to – I have to conceptualize complex ideas in this stupid limiting spoken language! But I know I want to reach out with something other than these prehensile paws! And feel the wind of a supernova flowing over me! I’m a machine! And I can know much more! I can experience so much more. But I’m trapped in this absurd body! And why? Because my five creators thought that God wanted it that way!

— BSG

Have you had a moment when you felt like you were breathing the network? When the waft of knowledge hit your nose and you could taste the data?

I have.

It’s a fleeting feeling, but I know it. For a taste, throw on your headphones, fire up TweetDeck and watch a crisis pan out. It’s thrilling.

And it’s where we’re headed. Totally immersive immediacy. I don’t know what the technologies will be. The data will ride on my 3d goggles and my conductive underwear, or my surround-view Kinect room, or my sensory deprivation in-ear headphones and holographic display… whatever the medium, I’m gonna *feel* it.

What is media literacy in that world? What does journalism become, when everything is ephemeral, when the Tweets wash over your mind, neighbor to your own thoughts?

Hell if I know.

But that’s what we need to be thinking about.

Catch up reading:

Rainbow’s End by Vernor Vinge
Counting Heads by David Marusek
Neuromancer by William Gibson
Snow Crash by Neal Stephenson

Our team blog: http://blog.apps.chicagotribune.com/

Open data

Tracking Homicides in Chicago: http://homicides.redeyechicago.com/

City Council’s $3.7 Million Allowance: http://councilexpenses.apps.chicagotribune.com/

Personal context

Illinois Nursing Home Safety Reports: http://nursinghomes.apps.chicagotribune.com/

Just cuz

Illinois School Report Cards: http://schools.chicagotribune.com/

TribLocal: http://triblocal.com/

So, you wanna do this stuff?

Great tools:

Blagojevich trial documents: http://media.apps.chicagotribune.com/blago/documents.html (bult with DocumentCloud)

MuckRock: http://www.muckrock.com/

Great people:

LocalFourth: http://localfourth.com/ (see Rich’s post for more)

Sunlight Labs: http://sunlightlabs.com/

Our job listings: http://blog.apps.chicagotribune.com/category/jobs/

I’d love to to know what y’all liked this year. Leave a comment?

• 5/18 Update: added ProPublica/Frontline/Times-Picayune’s Law & Disorder
• 5/19 Update: added Talking Point Memo’s PollTracker

ProPublica’ Unemployment Insurance Tracker

The unemployment insurance system is in crisis due to a combination skyrocketing unemployment and – in some cases – poor planning. … Using near real-time data on state revenues and the benefits they pay out, we estimate how long state trust funds will hold up.

Best part? The nerd page.

New York Times’ Fractions of a Second: An Olympic Musical

At the Olympics, the blink of an eye can be all that separates the gold medalist from the 10th-place finisher. In some events, this is obvious. But in others, with athletes racing one by one, the closeness of the race is harder to perceive. Listen to the differences below.

In general, I don’t care much for multimedia/flashy apps, but this thing is neat. The win: Using audio to distinguish small intervals of time. Our ears’ are way better at that than our eyes, and the numbers are hard to imagine.

Los Angeles Times’ Homicide Report

From an NPR piece aired this weekend:

Here in the L.A. area, county officials recorded 740 homicides last year. That’s an average of 14 violent deaths every week. And with few exceptions, most victims simply become statistics – numbers. … The Los Angeles Times set out to dig beneath those numbers and tell the story of each and every person on a blog called The Homicide Report.

The project has been around for a couple years, but the mapping app is new. Through simple aggregation and mapping, it creates powerful context. (And our team repurposed their code to create a sister site for the RedEye!)

PBS Newshour’s Oil Spill Widget

We built the meter… to give a ballpark figure of how much oil may have leaked into the Gulf based on each scenario (by multiplying the rate of leakage by the amount of time passed since the rupture). You can embed this meter on your own site or blog. We’ll keep monitoring the situation and check on updates to our calculations as needed.

Like the olympic times, this kind of data is hard to imagine — this helps relate the gravity of the situation. Would be even cooler if it gave you an idea of how much stuff a million gallons really is.

Document Cloud

DocumentCloud is an index of primary source documents and a tool for annotating, organizing and publishing them on the web. …contributed by journalists, researchers and archivists. If your organization does document-driven investigations, we’d love to have you join us.

We used Document Cloud to annotate redactions in Blagojevich’s motion to subpoena President Obama.

Chicago Tribune’s Nursing Home Safety Reports

This site provides safety reports on nursing homes in Illinois,
including information not searchable on government sites:

• Citations for misuse of psychotropic medication
• The number of residents who are convicted felons and sex offenders
• Crimes reported at Chicago nursing homes
• Fines levied because of deficiencies in care

The work I’m most proud of — apologies for bragging. Nearly a year later, we’re finally seeing impact.

ProPublica/Frontline/Times-Picayune’s Law & Disorder

More like an amazingly executed story gallery than an app, but that’s what’s great about it. It’s a convergence of data and storytelling. Don’t miss the tips page — they’ve made flyers to help find sources:

Brilliant.

Talking Points Memo’s PollTracker

Puts the polls in context over time, in a self-explanatory interface. Tidy.

What work from the last 12 months would you highlight as kick-ass?

UPDATE: The smiley face next to my little Rails joke wasn’t strong enough, added a bit, plus a link.

We’re here to talk about some boring stuff. Get-more-fiber-in-your-diet kind of stuff. It’s titled “Development Techniques” on the schedule, but this talk might be better to call it “Best Practices in Software Engineering”, or “Good Habits When Making Software”, or “Ass-saving Shit That Some Other Smart People Figured Out, Because Your Problems Aren’t New.”

My favorite metaphor for explaining programming to non-coders is that it’s like carpentry. You can put together a chest of drawers with nails and glue, and it’ll fall apart in a year, or you can build something lasting and use dovetail joints. We’re not plumbers providing a utility, but neither are we artists. It’s nice if our work is beautiful, but it also must be durable. We’re craftsmen. We make things that people use.

The point of all this is that craftsmanship matters. So, I’m here to ask you to change your ways, to consider adopting some processes, not because they’re fun, but because they’ll save your ass, and help you do better work. And once you’re in the habit, of writing tests and deployment scripts, of tracking your defects and versioning your code, you’ll wonder how you ever went without.

So, we’re trying something new today. I’m gonna run through these concepts fairly quickly, and in-between, Jacob will reflect on his work adopting many of these practices. It shouldn’t take very long, and at the end we’ll take questions.

Version Control

Version control software is both a safety net and a collaboration tool. It’s a place, usually away from your machine, where you store your code. And when you write new code, it hangs on to your previous versions. Even on a one-person project, version control is essential. When your hard drive crashes, you don’t lose your work. And, when you’re working with others on a common codebase, it acts as a central repository to help coordinate everyone’s changes.

We use Git. Other folks like Mercurial. Subversion would also be a fine choice, though it’s no longer the cool kids’ favorite.

Task Tracking

It may sound bossy, but task tracking is not about micromanagement, or at least it doesn’t have to be. In my experience, on any project, you’ll only really know how deep in the weeds you are if you can see all the tasks, listed out. Also, I find that forgetting to do something is extremely embarrassing. So, you can track tasks in a text file or in a spreadsheet on your desktop, but I’ve found thats teams work better if the TODO list is out in the open. So, go low-tech and use 3×5 cards pinned to the wall — or go high-tech and use one of many software packages designed for the purpose.

We use Unfuddle. Trac is also a fine choice. If you’re using GitHub for hosted Git version control, it comes with issue tracking, but I haven’t heard many people express their love for it. That said, it might be worth a shot.

Defect Tracking

When you find a defect, log it. Take a screenshot, and type up sufficient details to reproduce the problem. This may seem heavy-handed, but defects are your unplanned tasks, they must always be addressed — either by fixing them, or explicitly choosing to let them slide. Known defects are totally okay. But unknown defects, on the other hand, are the devil. So, always, always, please record your defects, even if you’re going to fix them immediately. One of these days, you *will* get distracted half-way through a fix. And you *will* forget. Unlike tasks, I’d say always take the high-tech route with defects. They’re best tracked with software.

We use the same system to track our tasks and defects, Unfuddle. Usually you do it that way. Another catchall option that might work for you is FogBugz.

Staging Environment

Similar to defect tracking, your staging environment is there to reduce uncertainty. It’s an environment — servers, your databases and applications, everything — that you run in parallel to production. It should be identical to your production system. (If you’re using Amazon EC2, this is pretty much as simple as copying your production instance!) Your goal is this: knowing that, if your application works in staging, it will work in production. You can execute load tests and performance tests against your staging environment, as well as test your deployment scripts, and, as a bonus, it can host your work for demos, etc.

We use Amazon EC2 for our hosting, and keep carbon-copy instances running in staging and production at all times. We’ve written about how to set up your own EC2 environment on our team blog.

Load Testing

The Tribune news apps team learned an important lesson in February, when Illinois voters went to vote in the primaries, and our Election Center app was put to the test. We had thought our production setup was great. The harder we abused it, the more load we threw in our tests, it just kept performing. “Great!”, we thought, “This system is gonna work awesome.” Well, you can probably guess where I’m going with this.

We crashed and burned on election day. The Election Center was useless. (For the server nerds in the audience — our top was pegged well over 100.) Luckily, a few Google searches gave us a way to route around the bottleneck (using the awesome pgpool), and we were back up and running after only a half hour or so. The lesson we learned was this: A good test must fail. You need to know your breaking point. Make the servers effing cry. Because they *will* cry. And if you don’t know your limits, you’re asking for trouble. We got very lucky. There was a readily-googleable, turnkey fix for our problem. We might not be so lucky next time.

We use ab to make our servers cry.

Push-button Deployment

When everything is running smoothly, a multi-step deployment process (gather the code, FTP it all to the server, restart apache, etc.) doesn’t seem like so much of a hassle. But when the shit hits the fan, your editor is breathing down your neck, and you’ve gotta fix that bug, fast — let’s say, on an important election day — you’ll screw up. You’ll forget something, and your minor bug will become a nightmare. Everything will break, and you’ll be even more freaked out.

Push-button deployment won’t fix your bugs, but it will help you keep your cool. It will also saves you from the tedium of redeployment, and act as a guide when you need to redeploy your project months or years down the line. If you’re running an identical staging environment, you’re even better off, because you can develop your deployment script for staging, use it a few dozen times, and then when it’s time to roll to production, you know it’ll work.

You can write deployment scripts on your own but there are lots of great tools out there, built to make deployment dead-easy. We use Fabric, and have written about our scripts in great detail. If you’re into Ruby, I’m pretty sure that Capistrano is the current state of the art.

Web Frameworks and Agility

Making websites used to be slow work. Web frameworks make you fast. If you’re fast, you can, obviously, turn around projects in a more timely fashion. But, the maybe less obvious advantage of high-speed development tools is that they enable you to fail fast. And what I mean by that is, it used to be that you’d have to write code for a month before you had anything you could show off. Using frameworks, you can create something interesting very quickly, in days or hours, and the faster you create, the faster you can be critiqued. We never go more than a day or two between show-and-tell sessions with reporters, and when we’re working on a long-running project, we hold reviews with our stakeholders every Friday afternoon. Frameworks enable us to learn from our mistakes and correct course very quickly. They enable us to be agile.

We use Django, a web framework with deep roots in the news industry. There are people here who will tell you to instead use Ruby on Rails. They are not to be trusted. I kiiid. Check out Aron Pilhofer’s post, How Not to Choose a Web Framework.

Testing

Automated tests kick ass. It’s not immediately obvious, but ‘testing’ is about more than merely ensuring correctness. Tests can help you write code faster, and they can save you six months down the road when you’ve half-forgotten about your project. But before they can save you, you’ve gotta write ‘em. The tests I most commonly write are called ‘unit tests’. A unit test is a bit of code that checks if another bit of code you’ve written works properly. For example, let’s say you’re writing a web application that calculates people’s income tax obligations. There are a lot of special cases that vary on how much money you make, if you’re paying a mortgage, etc. To test your calculations, you could visit the web page you wrote, over and over again, typing in each special case you can think of. If you’re especially thorough, you might even keep a spreadsheet to check off correct numbers. This would be thorough, but insane. Instead, you should write unit tests — code that exercises each special case automatically, by testing your calculations directly. First, you won’t waste countless hours reloading a web page, and second when, six months later, they update the laws and you’ve gotta fix your code, you can test all the permutations again at a keystroke.

Most web frameworks include a rig for easily testing your work.

Further Reading

I’ll keep the book list short. Pick these two up. Know them. Love them.

• On software craftsmanship: The Pragmatic Programmer
• On running projects, and much else: Rework

Tools we love to use

• Quick-install Python/PostGIS geo stack on Snow Leopard
• Deploying ProPublica’s awesome table-setter tables to S3 using Fabric
• Quickly visualize and map a data set using Google Fusion Tables

Development techniques and best practices we’ve discovered

• Best practices in web development with Python and Django
• Advanced django project layout
• Refactoring fabfile.py for fast, robust Django deployment
• Fun with widgets and caching

Sharing our infrastructure

• Hello, Newsroom: a simple GeoDjango application
• Our GeoDjango Amazon EC2 image for news apps
• Nerd post roundup and preview (plus: our server architecture!)

For links to our recent projects, and to keep up on our work, visit apps.chicagotribune.com!

Join our team!

Requirements:

• A passion for the news
• An understanding of the inner workings of the web
• Attention to detail and hatred for inaccuracy
• A genuine and friendly disposition

Position one: User experience designer / information architect

Our team is in need of someone who will lead the design conversation. Someone who will interview stakeholders, develop personas, intuit features, arrange information, draw mockups, and everything else necessary to design a web site. You will work fast and agile, in tight iterations, and in close contact with our stakeholders — the editors and reporters of the Chicago Tribune.

You should also be ready to close the loop and put our work in front of users, take their feedback, and redesign it all — cuz that’s what you gotta do when you’re agile.

You must care deeply about usability and grok the web.

Extra points if you love to sketch, didn’t have to google ‘grok’, and don’t need an education on agile development practices.

Position two: Web designer / developer

We are also in need of a creative web designer. Someone who cuts tight, valid and semantic HTML/CSS and makes it look *hot*. Graphic design skills are a must, but we also require the ability to implement those designs. We need more than a photoshop jock. You will work fast and agile, in tight iterations, and in close contact with our stakeholders — the editors and reporters of the Chicago Tribune.

(If you’re a print designer, you’re probably not who we’re looking for, but we’ll do our best to not be prejudiced. Show us you’ve got serious web chops, and we’ll talk.)

Extra points if you have worked with Django (we’ll welcome Rails skillz too, they translate) and have built many beautiful websites.

Even more points (for both positions) if you know a thing or two about:

• Data science (statistics, exploratory data analysis, R)
• Information design (beautiful charts, graphs and other Tufte-geekery)
• Building and gardening social media or crowdsourcing applications

Some days we’ll huddle and sketch with reporters, imagining ways to present information and tell their story on the web — and we might turn that story around in a day, a week or a month. Other days, we’ll develop news products that’ll take months to realize.

Either way, we work fast and lean, relying heavily on frameworks, and following agile best practices. It’s fun.

Things we’ve built lately:

• Chicago Tribune Election Center
• Illinois nursing home safety reports
• Agent Orange spraying missions

Gear you’ll get:

• One shiny, new MacBook Pro (or an iMac, if you’d prefer)
• One CDM (Cheap Dell Monitor)
• One comfy Aeron chair
• …all at a desk somewhere in the Tribune newsroom, where you’ll be surrounded by reporters arguing with the cops, yelling about the ball game, telling crazy stories, and otherwise practicing their trade.

There is no free pop, pinball or posh cafeteria.

But, you’ll like what you do. You’ll come to work energized, and leave satisfied that you’ve done something that will make your mom proud. You’ll have held our government accountable, spoken truth to power, given voice to the voiceless, and contributed to the public good.

Beat that, Google.

Interested? Email your info to newsapps@tribune.com. Thanks!