Kevin Schaul

TweetThis post was first published at journalists.org.

There’s nothing less funny than listening to a journalism professor joking that we’re all in this field because we can’t do math. Some of the best journalism being done today only exists because journalists overcame their fear of numbers and dug deep into the data.

Take the L.A. Times’s series on 911 response times. An analysis found stark disparities in the response times of emergency vehicles, and it produced journalism with real impact. Not bad for a bit of math.

Or look at USA Today’s diversity index. Journalist Phil Meyer and Paul Overberg, the paper’s database editor, invented a way tonumerically compare racial and ethnic diversity — no small feat back in 2001. This index opened up a wealth of unreported stories, and gave measurable evidence to those we only believed anecdotally.

You might say, “But these are both CAR stories,” and you’d be right. But we are all computer-assisted reporters. The moment a news organization sheds the backwards thought that only a select few can understand data, millions of uncovered stories will be discovered.

In a recent talk, Ryan Pitts, senior editor for digital media at The Spokesman-Review, and Jeremy Bowers, news applications developer at NPR, walked through how indexes like USA Today’s can be created to fit your beat. How does this business tax proposal compare to previous laws? Create an index for it. Which college is the most cost-effective for students? Ditto. If Nate Silver of the New York Times has taught us anything, it’s that people trust data over a reporter’s intuition.

Of course, not all questions can be answered with indexes. Statistics provides the tools to figure out information we haven’t even considered yet. The more mathematically savvy journalists you have in your newsroom, the more groundbreaking journalism your company will produce. (And yes, that could be quantified.)

Recently Chase Davis, new assistant editor of interactive news at the New York Times, explained five algorithms with huge potential that journalists have not yet explored. Want to know which politicians in your state are the most similar? Run a nearest neighboranalysis. Need to classify thousands of bills into clean categories? Try a random forest algorithm, and let the robots do the work for you.

The more we become familiar with these sorts of solutions, the more stories we can pitch. Editors love reporters who find new angles on our world, and data-driven work is no exception.

And here’s the best part: The hard work has already been done for us. Open source toolsexist to ease the computation efforts of these statistical models. We’re on the brink of using technology to better understand subjects as complex as campaign finance and how elections are won. Mathematicians have produced loads of information analysis techniques that are just waiting to be taken advantage of by journalists. We don’t need to be experts in statistics to find answers to our interesting questions.* We just have to get over our fear of numbers.

If you can say “algorithm” with a straight face, I’ll bet there’s a job out there for you. And don’t let your journalism professors get away with their cheap math jokes. The times have changed.

*Of course, it’s easy to lie with data. Be sure to run your work by someone who does know what they’re talking about before publishing.

TweetQuestioned on The New York Times’s use of D3 for graphics, even though IE 7 and 8 do not support it, Amanda Cox gave this response:

“There has been little resistance from our readers to doing awesomeness.” – @nytgraphics‘s Amanda Cox, on not always supporting IE #nicar13

— Kevin Schaul (@kevinschaul) March 1, 2013

I can’t wait to work with these people.

TweetIn yesterday’s New York Times, I ran across an unusual opinion piece on gerrymandering. Sam Wang, the column’s author, made his argument with data.

“Using statistical tools that are common in fields like my own, neuroscience, I have found strong evidence that this historic aberration arises from partisan disenfranchisement,” Wang said. A graphic ran alongside the piece further explaining Wang’s statistical findings.

Of course, statistics (in traditional and visual forms) can be a wonderful tool to removing emotion from reality and giving us a raw version of what’s really happening. But we also know that statistics and visualizations can be used to mislead.

So, do statistics and graphics belong in the opinion section?

My initial reaction is that they absolutely do. The public already takes an extremely critical view of anything published by a news company. Readers should treat a data-driven piece no differently. Opinion writers have the duty to promote worthwhile discussion and encourage change, and the best do this by admitting anything worth debating is not black-and-white. Statistical methods are powerful and must be used to enhance this discussion, not to alter truth.

I see it as an ethical duty of opinion writers to be fair about their data, just as they shouldn’t mislead with anything they write. For complete clarity, I would have liked to see both Wang and graphics editor Bill Marsh publish their methodology in a related post.

The technique is young, but it isn’t likely to go away. I’d love more discussion on this.