A quick primer on making software — best practices, tools and further reading

ul.toolslist li {line-height:2em;}
ul.toolslist li ul li {line-height:1.5em;font-size:.8em;}

This is more-or-less a script for an un-un-conference talk I’ll be giving shortly at ONA09.

“Imagine a news organization with only writers, and no editors. They might manage to crank out some successful stories, but without editorial controls, the failure rate would be astronomical.”
Me, a couple of months ago.

Why we do this

You don’t adopt processes because they’re fun, you adopt them because they have special ass-saving properties. Doing it the right way may seem heavy, micro-manage-y, but when the process sings, the unbearable weight of uncertainty is lifted from your shoulders. This is freedom through tyrrany.

A seasoned developer won’t find much of what follows particularly interesting. This is elementary, but it’s stuff that seemed worth talking about…

A few baseline requirements for anyone making and releasing software

Version control

Version control software is both a safety net and a collaboration tool. It’s a place, off your machine, to keep your code, and when you update the code, it keeps your previous version(s). So, even on a one-person project, it’s essential. When your hard drive crashes, you don’t lose your work. And, when you’re working with others on a single codebase, version control gives you a central repository to coordinate everyone’s changes.

Task tracking

Task tracking is not about micromanagement (or, at least it doesn’t have to be). You’ve gotta be able to see the tasks on the docket so that you can know how deep in the weeds you really are. Also, forgetting to do something is really embarassing. You can track tasks in a spreadsheet, but that’s not very visible to the team. Instead, go low-tech — 3×5 cards pinned to the wall — or high-tech — with one of many software packages designed for the purpose.

Defect tracking

When you find a problem: log a defect. Take a screenshot, and give sufficient details to reproduce the problem. Defects are your unplanned tasks, they must be addressed — either by fixing them, or choosing to let them slide as a known defect, which is totally okay. Unknown defects, on the other hand, are the devil. Always, always record your defects, even if the very next thing you’re going to do is fix it. You *will* be distracted. You *will* forget. Defects are pickier than tasks, and are best tracked with software.

Staging environment

Like defect tracking, having a staging environment is about reducing uncertainty. It’s an environment running in parallel to production, set up as identical as you can make it to the production system. (If you’re using Amazon EC2, this is pretty much as simple as copying your production instance!) Your goal is this: knowing that, if your application works in staging, it will work in production. You can run load and performance testing against your staging environment, test your deployment scripts, and, as a bonus, it makes for a nice place to demo your work before it’s finished.

Push-button deployment

When everything is running smoothly, a multi-step deployment process (grab the latest code, ftp it all to the server, restart apache, etc.) doesn’t seem like so much of a hassle. But when the shit hits the fan, folks are freaking out, and you’ve gotta redeploy, half-drunk on a friday night, you’ll screw it up. You’ll forget something, and your minor bug will become a total clusterfuck. But if you’ve got push-button deployment, you can’t miss. If you’ve got a identical staging environment, you’re even better off, because you can develop your deployment script for staging, use it a few dozen times, and then when it’s time to roll to production, you know it’ll work.

The tools we use

Further reading (please, read further!):

Hacker wanted: Code in the public interest, save journalism, in sunny Chicago, Illinois

UPDATE: We’ve filled the position, but may be hiring more soon.
If this looks like your dream job, please send an email anyway. 🙂

Cross-posted from our new team blog:

We’re looking for a great hacker to join our team at the Chicago Tribune.


  • A passion for the news
  • An understanding of the inner workings of the web
  • Attention to detail and hatred for inaccuracy
  • A genuine and friendly disposition

And, of course…

  • Bad-ass programming skills and a love for the craft of making software

Tools we use (and thus, tools we hope you might know a thing or two about — if you don’t, that’s okay, but please explain yourself):

  • Python
  • Django
  • PostgreSQL
  • PostGIS
  • Ubuntu Linux + Amazon EC2

Our team is composed of generalists: We all write GUI code, mine data and manage servers. You ought to be equally comfortable wearing many hats. That said, we’ve all got our specialities, and would love to find a team member with a superpower which none of us already possess. Something like…

  • Data science (statistics, exploratory data analysis, R)
  • Information design (beautiful charts, graphs and other Tufte-geekery)
  • Maintaining high-performance web sites (cuz we’re gonna get serious traffic)
  • Building and gardening social media or crowdsourcing applications

You’ll work closely with reporters on the investigative and city desks, helping them research and present their work. Sometimes you’ll be screen scraping, mucking with data, visualizing and exploring information, and seeking truth. Other days, we’ll huddle and sketch with reporters, imagining ways to present information and tell stories on the web.

Sometimes we’ll knock out an application in a day, other times it’ll take a few weeks. Either way, we work fast and lean, relying heavily on frameworks, and following agile best practices. It’s fun.

Things we’ve built lately:

Folks you’ll work with:

Gear you’ll get:

  • One shiny, new MacBook Pro
  • One CDM (Cheap Dell Monitor)
  • One comfy Aeron chair
  • …all at a desk somewhere in the Tribune newsroom, where you’ll be surrounded by reporters arguing with the cops, yelling about the ball game, telling crazy stories, and otherwise practicing their trade.

There is no free pop, pinball or posh cafeteria.

But, you’ll like what you do. You’ll come to work energized, and leave satisfied that you’ve done something that will make your mom proud. You’ll have held our government accountable, spoken truth to power, given voice to the voiceless, and contributed to the public good.

Beat that, Google.

Interested? Email your info to newsapps@tribune.com. Thanks!

Sex offenders: Your tweets (and LinkedIn and TimesPeople) are now a felony

Required qualifying statement: if you’re a sex offender, you’ve likelypossibly (as pointed out by Asim, a recent piece in The Economist suggests that sex criminals in the U.S. are often victims of our screwed-up laws) done something very bad, and of which I do not, in any way, approve. That said, I’m pretty sure you still have a few rights…

As reported in the Chicago Tribune today, social networking is now a felony for many Illinois residents:

One law taking effect Jan. 1 makes it a felony for registered sex offenders to use social networking sites, a move aimed at taking another step toward shutting down an avenue of contact between an offender and victim.

The bill
defines social networking as such:

13 		    (h) "Social networking website" means an Internet website
14 		containing profile web pages of the members of the website that
15 		include the names or nicknames of such members, photographs
16 		placed on the profile web pages by such members, or any other
17 		personal or personally identifying information about such
18 		members and links to other profile web pages on social
19 		networking websites of friends or associates of such members
20 		that can be accessed by other members or visitors to the
21 		website. A social networking website provides members of or
22 		visitors to such website the ability to leave messages or
23 		comments on the profile web page that are visible to all or
24 		some visitors to the profile web page and may also include a
25 		form of electronic mail for members of the social networking
26 		website.

Sure, it seems right to stop violent criminal perverts from poking around MySpace, but every damned site on the web is integrating social networking features nowadays. Is it offensive to the public interest if a sex offender shares an article on TimesPeople? What about LinkedIn? Do sex offenders not deserve a place to post their resume?

Or, what about a social network devoted to sex offenders trying to rehabilitate? If there isn’t a Ning for this already, there sure oughta be.

This is bad legislation. Sex criminals have rights too, and this law effectively bans them from the Web.

Kick Ass News Apps! — projects to inspire journos

To introduce ourselves and our skills to the Trib newsroom, Joe and I showed off some news applications we love, and that we hope will inspire the journalists here to think about telling their stories in new ways online.

For the folks who missed the show, here’s a quick rundown of what we talked about. (I am sad to say that there is no way to serve refreshments through the web, so to get the full experience, you’ll have to get your own punch and pie.)

PolitiFact’s Obameter

Politifact's Obameter

Reporters and editors from the [St. Petersburg Times] fact-check statements by members of Congress, the White House, lobbyists and interest groups and rate them on our Truth-O-Meter. We’re also tracking more than 500 of Barack Obama’s campaign promises and are rating their progress on our new Obameter.

  • Brian sez: It hits the sweet spot between software and old-school reporting. Hacker journalism at its best.
  • Joe sez: Demonstrates the power of the web to provide context over time beyond each day’s story.

Tampa Bay Mug Shots

Tampa Bay Mug Shots

Our goal is to provide a complete profile for individuals booked into jail in Pinellas, Hillsborough, Manatee and Pasco counties. A complete profile on Mug Shots constitutes: name, photograph, booking ID, height, weight, age, gender, eye color, birth date, booking date and booking charge.

  • Brian sez: It’s tabloid, trashy stuff in a great-looking package. Pretty hot for a system that shows off public records.
  • Joe sez: I’m not sure how I really feel about this app, but it is a great example of making bulk data accessible to the general public.



ChangeTracker watches the White House’s web site so you don’t have to. Whenever a page on whitehouse.gov changes, we’ll let you know — via E-mail, Twitter, or RSS.

  • Brian sez: This is my project, so I’m partial, but… It’s a simple concept with many interesting uses — as both a reporting tool, and as a publishing device. Plus, it’s free and dead-easy to set up your own.
  • Joe sez: Tools like this protect us from the risk of information going down the “memory hole.”



Some senators like to filibuster and keep the majority from having their way. You might think they’re heroes. Or jackasses. Either way, they’re worth keeping track of.

  • Brian sez: It’s tightly focused site that does a great job explaining an issue that most folks don’t understand.
  • Joe sez: This one was a winner in Sunlight’s Apps for America contest. It would be easy for us to tap into the same data about legislators, bills, and votes that feed this one.

Represent and Repsheet


RepSheet lets you…

  • look up your elected representatives…
  • see the political zones you live in…
  • and track news about your reps.

  • Brian sez: The Times’ Represent and Windy Citizen’s loving rip-off, Repsheet, are, like Filibusted, tightly focused and explain something most folks don’t understand — in this case, the overlapping districts of representation. And they give you an easy way to follow news on what your reps are up to. Kinda hard to believe how difficult this was before, eh?
  • Joe sez: With the amount of information on the web, we need more tools like these that help people focus on what matters most to them.

Investigate your MP’s expenses

Investigate your MP's expenses

Join us in digging through the documents of MPs’ expenses to identify individual claims, or documents that you think merit further investigation. You can work through your own MP’s expenses, or just hit the button below to start reviewing.

  • Brian sez: How would *you* search through a half million pages? And the UI is wonderfully simple.
  • Joe sez: This app does a great job of keeping on the story while it’s current. Its release is an attention-grabber and can help the Guardian investigate the data even if the public’s participation is minimal or inaccurate. This Nieman Labs article provides some good lessons learned from Simon Willison, the application developer.

Many Eyes: Word tree and US Gov’t Expenses chart

Many Eyes: Word tree

Many Eyes is a bet on the power of human visual intelligence to find patterns. Our goal is to “democratize” visualization and to enable a new social kind of data analysis.

  • Brian sez: Many Eyes is a fun kit of visualization tools that are easy for anyone to populate with data and embed in a story. They’ve got maps, charts, word trees and all sorts of other neat toys to play with.
  • Joe sez: Not only are these tools a great way to provide basic data visualization, but most of them also provide readers with the ability to explore different views of the data.

Names, Lists, Photos, Stories – California’s War Dead

Names, Lists, Photos, Stories - California’s War Dead

Military deaths in Iraq and Afghanistan, 2001-Present

  • Brian sez: It’s a simple application that uses data to tell a story from many angles. If you’re a parent, maybe you want to see how many kids the soldiers had, or maybe you want to explore based on where they were from. Simple, but powerful stuff.
  • Joe sez: By collecting information about all of the dead, the Times creates a richer story center around which they can also aggregate their original coverage.

Know thy Congressman

Know thy Congressman

“KTC” is a bookmarklet that displays an abundance of political and biographical information about current members of the Senate and House of Representatives.

  • Brian sez: It’s a totally novel tool, a great use of public data, and incredibly useful. Plus, they probably coded it in a weekend. Love it in every way.
  • Joe sez: Another great example of how the web makes it so much more possible to provide background and context for stories.

Watching the Growth of Walmart Across America

Watching the Growth of Walmart Across America

Yesterday I quickly put together my own Walmart growth video using Modest Maps. It has the usual mapping features – panning and zooming – while you watch Walmart spread like wildfire. It starts out slow with the first location in Arkansas in 1962 and then spreads vast in a hurry.

  • Brian sez: Eye candy, for sure, but damn tasty eye candy.
  • Joe sez: Visualization across space and time tells parts of the story better than words possibly could.

ONA09 un-conference session proposal: The craft of making software — Anyone interested?

UPDATE — The folks at ONA have announced that they’ll provide rooms for un-conference talks! Woot! But if there’s more need than space, it’ll be up to a vote, so *please* get there early and vote me up! Hope to see you there!

The schedule for ONA09 is jam-packed with shiny stuff — social networks, mobile tech, they’ve even got Leo and Ev! Great. But the reality is that Twitter will not save the news, just like chrome rims can’t save General Motors.

We can talk about technologies, tools and innovators all weekend long, but it won’t help if news organizations don’t understand the basic principles of software development. So, if anyone out there is interested, I’d like to arrange an un-conference to talk about some un-shiny, boring-ass shit: software development methodologies.

Topics of import we might address:

  • Version control
  • Task and defect tracking
  • Goals, use cases and designing with your audience in mind
  • Working iteratively and being agile

Code is not something you can slap up like wallpaper. Making software is a craft. It requires discipline and skills far beyond a superficial awareness of the technologies available. At every moment of the process, from conception to release, there are right and wrong ways to make software.

Imagine a news organization with only writers, and no editors. They might manage to crank out some successful stories, but without editorial controls, the failure rate would be astronomical. From what I’ve learned in my (admittedly brief) time in this industry, this is the state of software development at newspapers — it’s failure-ridden, amateurish and ad-hoc.

Let’s do it the right way

Over the years, lots of clever people have studied the craft of software development, and come up with battle-tested tools, best-practices and processes to reduce the failure rate and better-ensure success. I learned a thing or two about these methods in my previous life, and would love to share.

So, I’d like to set up an un-conference session. We’ll get a room and a projector and talk process. Who’s interested in attending? What topics would you like to see addressed? Would anybody else like to present?

(If there’s no response, I’ll shut up and go back to work — but if I’ve convinced you, please leave a comment. No comments, no un-conference.)

Old friends! Ripoffs! Hateful comments! — A media blitz roundup

Invisible airwaves, crackle with life…

I don’t write about it much here, but I love the radio. NPR is my primary daily news source. So, I was totally geeked when Here and Now asked me to be on their program. And through the magic of editing, I think I managed to sound pretty alright!

(You know how, when someone gets under your skin, but you’re so worked up all you can say is “Oh yeah? Well, your momma!”? And then later, when the moment is gone forever, your brain catches up and comes up with all sorts of erudite, polysyllabic arguments? Radio is like that.)

Best part about the radio bit? All the old friends who emailed me after they heard the program, including one of my favorite high school English teachers. Dunno if/when the web will have that kind of reach. Love.

“…part of a hegemonic institutional perspective that is glaringly…”

Back online, my Hackers wanted! bit on O’Reilly Radar got a lot of attention, mostly hating on journalism and grumbling about needing a bachelors degree before pursuing a masters degree. But I’m hopeful that the message got through to a few programmers who want to make the world a better place.

In retrospect, I should have probably explained the importance of journalism in a democratic society, and ceded that yes, journalism is mostly broken right now, but this is our opportunity to fix it. Hindsight. Le sigh. If you’re into it, check the comment threads on the original post, and the nerdy aggregators that picked it up:

S’pose if you’re not pissing somebody off, you’re doing it wrong. Right?

Rich Gordon, the fella who decided it was a good idea to bring coders to J-school, wrote a response on Idea Lab including an interview with my new boss, Digital Editor at the Chicago Tribune, Bill Adee.

I even got ripped off completely by Tech Crunch, where fellow Medill alum Leena Rao summarized my post and completely failed to mention where she got the brilliant idea — eliciting 85 comments. Thanks for not linking!! Love!

Mr. Wolfram, will you please show your work?

Wolfram Alpha is gonna be pretty fucking neat. From the creators of Mathematica, Stephen Wolfram and Wolfram Research, the system proposes to be “an authoritative source for data” from the sciences, social science, finance, pop culture, and damn-near anything else that can be quantified and calculated. Ask it for the volume of your favorite lunar crater, expressed in wheels of parmesano reggiano, and it’ll run the numbers. Math, FTW!

But as far as I can tell from Wolfram’s recent presentation at Harvard, Alpha will tell you the source of the data, and will sometimes show you an example formula for a type of calculation (how to figure the volume of a cylinder, etc.), but it will not show you the actual calculations necessary to arrive at the answer.

Instead, we’re supposed to trust the system. Alpha’s code is so complex, says Wolfram, that it would be vastly inefficient for a mere human to read the steps. The more sensible approach is to “try to do the best QA that we can.” (QA = quality assurance, a.k.a. testing) And I’m certain that he’s right — that thing has got to be a bear to test.

If your mother says she loves you, check it out

The problem is, if I want to use Alpha’s answers in a news article, scientific paper, or anything else requiring an authoritative source, I need to know that it’s right. Unless we arrive at some kind of universal agreement among scientists, academics, mathematicians, and everyone else that Wolfram’s creation is always right, I can’t believe Alpha’s answers if I can’t test them.

I’m baffled by this omission. Alpha can do extraordinarily complex work, but it wouldn’t pass high school physics. You’ve got to show your work! Scientists validate hypotheses by repeating experiments and comparing the results. “Trust me” is not an authoritative answer.

“An authoritative source for data” is misleadingly simplistic. If it does what it’s supposed to, Alpha is more like “an authoritative source for quantitative thought” — the ultimate almanac, complete with a staff of uber-geeks from every field worth researching, backed by machines capable of turning around complex calculations in microseconds.

It’s cool as hell.

It’s a pocket calculator for *everything*.

It’s the realization of Leibniz’s characteristica universalis.

But is it correct? Only one way to tell. Mr. Wolfram, will you please show your work?