Dot density maps are hard

A successful dot density map requires a specific data set. The points must be dense enough to be interesting, but not too crowded so as to overlap each other. Maps with multiple zoom levels must perform magic to correctly display points with optimal sparsity. Perceptually, dots indicate data at a specific location, which doesn’t bode well for census-block level datasets.

Hexagon binning can alleviate these issues. In hexagon binning, a grid of a hexagons is placed over the extent of the points, and the number of points intersecting each shape is saved with the grid. The grid can then be visualized based on this accumulation, enabling better comparisons between dense areas of a dataset. Since there are no points, there is no overlapping. The locations of the individual points are replaced by a less fine-grain grid, revealing the interesting – and many times more pertinent – trend data.

Many dot density maps suffer from crowding of points.
Binify uses hexagon binning to alleviate this pain and better display trends.

Performing hexagon binning on deadline isn’t easy. Mmqgis, a useful QGIS plugin, can help with the step of creating the grid, but it requires using a GUI and is finicky. It can’t easily be automated. And it certainly can’t end up in a Makefile, as we’d prefer all our data manipulation to.

Introducing: Binify – A command-line tool to better visualize crowded dot density maps. (That’s bin-i-FY, for you phonetics.)

Binify takes all the meticulous guesswork out of hexagon binning. Simply give the program a point shapefile, and it’ll output a calculated hexagon grid version of the data ready to be visualized.

Binify is available in the Python Package Index (PyPI) for simple installation. To get started, follow the instructions on GitHub. I built the tool with the simplicity to be used for exploratory analysis, and with enough customization to cover all needs. (Of course, it’s work in progress. If you have an idea, please open an issue on GitHub.)

While hexagon binning is not the ultimate solution for every dataset, it’s a viable option for many. I hope you’ll find Binify as useful as I already have.

There’s nothing less funny than listening to a journalism professor joking that we’re all in this field because we can’t do math. Some of the best journalism being done today only exists because journalists overcame their fear of numbers and dug deep into the data.

Take the L.A. Times’s series on 911 response times. An analysis found stark disparities in the response times of emergency vehicles, and it produced journalism with real impact. Not bad for a bit of math.

Or look at USA Today’s diversity index. Journalist Phil Meyer and Paul Overberg, the paper’s database editor, invented a way tonumerically compare racial and ethnic diversity — no small feat back in 2001. This index opened up a wealth of unreported stories, and gave measurable evidence to those we only believed anecdotally.

You might say, “But these are both CAR stories,” and you’d be right. But we are all computer-assisted reporters. The moment a news organization sheds the backwards thought that only a select few can understand data, millions of uncovered stories will be discovered.

In a recent talk, Ryan Pitts, senior editor for digital media at The Spokesman-Review, and Jeremy Bowers, news applications developer at NPR, walked through how indexes like USA Today’s can be created to fit your beat. How does this business tax proposal compare to previous laws? Create an index for it. Which college is the most cost-effective for students? Ditto. If Nate Silver of the New York Times has taught us anything, it’s that people trust data over a reporter’s intuition.

Of course, not all questions can be answered with indexes. Statistics provides the tools to figure out information we haven’t even considered yet. The more mathematically savvy journalists you have in your newsroom, the more groundbreaking journalism your company will produce. (And yes, that could be quantified.)

Recently Chase Davis, new assistant editor of interactive news at the New York Times, explained five algorithms with huge potential that journalists have not yet explored. Want to know which politicians in your state are the most similar? Run a nearest neighboranalysis. Need to classify thousands of bills into clean categories? Try a random forest algorithm, and let the robots do the work for you.

The more we become familiar with these sorts of solutions, the more stories we can pitch. Editors love reporters who find new angles on our world, and data-driven work is no exception.

And here’s the best part: The hard work has already been done for us. Open source toolsexist to ease the computation efforts of these statistical models. We’re on the brink of using technology to better understand subjects as complex as campaign finance and how elections are won. Mathematicians have produced loads of information analysis techniques that are just waiting to be taken advantage of by journalists. We don’t need to be experts in statistics to find answers to our interesting questions.* We just have to get over our fear of numbers.

If you can say “algorithm” with a straight face, I’ll bet there’s a job out there for you. And don’t let your journalism professors get away with their cheap math jokes. The times have changed.

*Of course, it’s easy to lie with data. Be sure to run your work by someone who does know what they’re talking about before publishing.

“There has been little resistance from our readers to doing awesomeness.” – @nytgraphics‘s Amanda Cox, on not always supporting IE #nicar13

— Kevin Schaul (@kevinschaul) March 1, 2013

I can’t wait to work with these people.

The first step to creating compelling graphics is to find interesting data. Box Chart Maker creates a very versatile type of chart that can represent almost any story involving numbers (so, yes, that’s almost every story ever, in some way). I’ll use last week’s vote to proceed on the confirmation of Chuck Hagel as Defense Secretary.

Now that we have some data, go to the Box Chart Maker site. Here, you’ll find an example graphic already created. That was easy! If you’re representing “Data title” and have 36 items, you’re done. Otherwise, we’ll want to customize these options.

Let’s start with the “Yes” votes. The motion won 58 yeas, so let’s represent that in our first chart. Fill out the form with the correct information (58 boxes, “Yes votes” as the label, colors as you see fit). Under advanced options, change the ID to “yes_votes” or similar. This will allow multiple charts on the same page. When the options look good, hit the update button, and you’ll see your chart appear in the preview area.

If you’re happy with the chart, click on “Show embed code” under the preview button. A text box will appear containing all the html/css code that represents your chart. Copy that, and throw it in a fresh text document.

Now, do the same for the “No” votes. When you’re happy, copy the code and put it under the “Yes” votes code. Save the file as an html file, and open it up in your browser. Here’s what mine looks like:

You’re done! Of course, you’d want to add context to the chart, such as that the motion required 60 votes to succeed. But for absolutely no hand-coding, that’s not a bad graphic. Paste the code directly into your blog or CMS, and you’ll have a nice web-friendly graphic to help explain your story. Not bad for a few minutes of work.

Pro tip: If you’ve got more time and some html/css know-how, it’s simple to enhance the output of Box Chart Maker. Here’s what I came up with after a few more minutes of tweaking:

Publish!

“Using statistical tools that are common in fields like my own, neuroscience, I have found strong evidence that this historic aberration arises from partisan disenfranchisement,” Wang said. A graphic ran alongside the piece further explaining Wang’s statistical findings.

Of course, statistics (in traditional and visual forms) can be a wonderful tool to removing emotion from reality and giving us a raw version of what’s really happening. But we also know that statistics and visualizations can be used to mislead.

So, do statistics and graphics belong in the opinion section?

My initial reaction is that they absolutely do. The public already takes an extremely critical view of anything published by a news company. Readers should treat a data-driven piece no differently. Opinion writers have the duty to promote worthwhile discussion and encourage change, and the best do this by admitting anything worth debating is not black-and-white. Statistical methods are powerful and must be used to enhance this discussion, not to alter truth.

I see it as an ethical duty of opinion writers to be fair about their data, just as they shouldn’t mislead with anything they write. For complete clarity, I would have liked to see both Wang and graphics editor Bill Marsh publish their methodology in a related post.

The technique is young, but it isn’t likely to go away. I’d love more discussion on this.

Now, this.

Looks like my algorithms class will come in handy after all.

In reading the exchange, I realized once again why I am in this field.

This post as much an archive for these emails as a place to append my own thoughts. My comments can be found after those of three of the most inspirational people in the industry – Michelle Minkoff, Heather Billings and Sisi Wei.

Michelle Minkoff:

Since my name was dropped a lot today, please allow me to give the kind-of-recent student perspective on a few of the points addressed since this thread took a turn for the very weird. Forgive my delay — was too busy living my coding dream at my awesome job that my education gave me the tools to do.

For those of you who don’t know me, I was a text-interested English major who fell, hard, for computer-assisted reporting, data visualization and coding. I graduated with a journalism masters’ degree from Northwestern in 2010, and have been working for the AP’s Interactive department in DC doing “my thing” for about a year now. I’m still at the beginning of my journey, but I’ve met some of my key goals pretty early on in my life, and I owe most of it to NICAR.

Warning: this is very, very long. This is what happens when passion and late nights combine.

1. I don’t know that I would did/do is anything so special. Much like we’re surprised when we teach, and others don’t see the light, I remember being surprised that many of my fellow students just saw it as an elective to complete their program. Frequent comments from my peers: “Are you excited because it’s Thursday, which means it’s ‘Geek Class’ day?” Me: AAAAAAAH, yes! Student: You actually like it, don’t you? How? Me: Who doesn’t?”

2. The practicality of “this can help you get a job” only appeals to some of us. I looked at things rather idealistically, and still do. I was intrigued by how these tools could make my journalistic analysis and presentation more interesting. Occasionally, I am contacted by classmates who wonder if they, too, could have an editorially-focused job if only, “I had paid more attention to that guy Derek or whatever, but his class was so boring”. As I point out to them, if you’re not into it as a weekly assignment, putting in 40+ hours a week on writing queries/code is a special kind of hell. But if you love it, it’s living the dream.

3. I know a few people from such classes who use it as an additional skill in a more traditional j-job. We may not hear about it as much, but that’s okay — we’re still making a difference. I think that’s more prevalent than the radical shift I experienced. One of my Medill colleagues agreed with a blog post where I wrote about the importance of being curious, although he maintained “you seem to have taken that madness to the limit”. I don’t do things halfway.

4. To Mark Ng’s comment: it wasn’t that I was already on the path, and Derek accelerated my career. I didn’t know that this was a thing at all. He showed me a whole new world. Yes, I figured out how I could make these skills work for me. It’s a shift in perception, but his role was immense. It’s also about a personality match — I remember some dry humor that fell flat on most ears, and I’d be almost falling out of my chair. Might have been different with someone else.

5. Derek wasn’t just a professor giving lectures in class. First of all, it meant a lot for him to be an active journalist and technologist. I was inspired by current projects, and I loved how he was able to combine the two fields. He also was not a programmer by official school training, which made it more approachable for me. I thought the coding world was out of this English major’s reach, until I learned he was a rhetoric major.

He answered copious emails and IMs, modified code and answered questions technical (what’s the file format if I need all the rivers in the US) and philosophical (why are my peers afraid of the command line?) and about the work history of such places (how does reporter collaboration work?) Another mentor once commented to me that much of what I was asking wasn’t about how to do things, but looking for support/reassurance. That’s an important and valid role for us to play in the role of people coming to this world. We can’t force interest, but we can turn a light bulb on.

6. When we talk about educators, be cognizant that we are all educators. Derek literally changed my life, and brought me to all of you (“You mean there’s a list where there are people besides you who talk about this stuff and do it for money? It’s a thing?” “Yes, Michelle.” Me: AAAAAAAH! I LOVE NICAR NOW!), but he is not the only one who gets credit for my educational growth. The enthusiasm, patience and role models that I’ve gotten from an internship (Ben Welsh, Ken Schwencke) and my real world time at my current org (Jonathan Stray, Troy Thibodeaux) are essential. I see them as educators, and getting that kind of support for one’s continued growth is super important. I WOULD NOT BE WHERE I AM WITHOUT EACH AND EVERY ONE OF THEM. A phrase our DC bureau chief refers to is an “intellectual home”. And that’s what I think we should be trying to give people, to entice them to the cause. That goes for classrooms and newsrooms.

7. To speak to Steve Doig’s point about how Brittny, Heather and myself are all women, and what that means for the tech gender stereotype, I’ll just say I’m getting more involved in understanding the gender-specific issues in tech, and I’d like to give a special shoutout to NICARians for being awesome. I have NEVER felt like I’m treated differently by you fine folks, and if anything, there are men among you more sensitive to gender issues than I am. And I have heard some horror stories.

You’ll notice my list of very close role models above features all men, but that’s never been weird to me. Doing journalism and tech is hard, you help make sure being of a minority gender isn’t a dissuading factor. That means a lot. Or maybe I just never got the memo women weren’t supposed to be techy. Guess it’s a little late now.

Let’s all be on the lookout for more people who get it. We need them to join the cause, and can help them feel at home. I know you’re out there, maybe lurking on this email list. Know that I, and hundreds of others, will help you make the jump if you find journalism and tech and data rewarding. We will welcome you home to NICAR, where I am proud to call the members of this organization my teachers, my friends and finally, though I can scarcely believe it — my colleagues.

(If you thought that was long, sympathy cards can be addressed to the role models listed above, for whom tomes of this length are a frequent occurrence.)

Heather Billings

Sisi Wei

My own story is only in the first act. I grew up loving journalism and technology, but I never had a good understanding on the two, much less how they could be combined. My first year in college became a sort of competition between journalism and computer science to see which meshed better with my goals in life. Luckily for me, I ended up on the blog of the Chicago Tribune’s news apps team.

Two things struck me.

First, what could be better than using computer science to push the boundaries of journalism? It was a win-win in that respect, and I figured I could be happy with some sort of coding-for-journalism as my day job.

But second, why were these people giving away their secrets? Journalism felt like a field where competition was its bread and butter, and here were the best of the best showing Joe Reporter how to build a population density map in Tilemill with remarkable detail. Here they were, giving college freshman the keys to future employment. To them, it was more than a day job.

Fast-forward to today, where I am just over halfway through my college years. It has been a wild ride, and I owe it all to you all.

Now, I’ll admit, my story isn’t as suspenseful as Michelle’s, Heather’s or Sisi’s. I am incredibly lucky to have found an area directly at the intersection of two of my greatest interests. And this isn’t a field where we’ll be able to learn everything we need to know in the journalism schools, especially during this transition period. But what more could anyone want out of a field than the NICAR folks have?

For me, this was the clincher. I don’t think I could walk away from the help-first-compete-later industry that we’ve fallen into. Where else can one find a 55-email-long thread, with many posts upwards of 500 words?

As for attracting more students to do what we do, it’s going to have to end up being at least an elective in our education. Don’t get me wrong: I have had great professors, and Seth Lewis has been a life saver. But without access to a Derek Willis, a Mark Ng or the forward structure of Medill, the skills I really craved had to be self-taught. As Robert Hernandez said, “Don’t wait for academia to determine what you need to know for modern journalism. Be proactive and find out by using digital media to help you learn those skills.”

At the University of Minnesota, I’ve begun gathering like-minded students who want to learn these skills but don’t know where to begin. We’re meeting weekly to teach each other what we won’t learn in the classrooms.

It’s an experiment, to be sure. But right now, I don’t see a better way.

@A_L Al Shaw – ProPublica
@aboutaaron Aaron Williams – L.A. Times
@ajdant Alastair Dant – Guardian
@albertocairo Alberto Cairo – University of Miami
@albertsun Albert Sun – New York Times
@AlexGraul Alex Graul – Guardian
@alykat Alyson Hurt – NPR
@andrewhaeg Andrew Haeg – Public Insight Network
@andrewphelps Andrew Phelps – New York Times
@AnthonyDB Anthony DeBarros – USA Today
@bposton Ben Poston – Milwaukee Journal Sentinal
@brianboyer Brian Boyer – NPR
@bycoffe Aaron Bycoffe – Huffington Post
@cephillips Cheryl Phillips – Seattle Times
@chasedavis Chase Davis – Center for Investigative Reporting
@ChrisLKeller Chris Keller – Madison.com
@dancow Dan Nguyen – SkiftNews
@DanHillReports Dan Hill – Northwestern University
@dansinker Dan Sinker – Knight-Mozilla OpenNews
@datatelling Jen Lowe – Open Knowledge Foundation
@derekwillis Derek Willis – New York Times
@eads David Eads – Chicago Tribune
@ericg Eric Gundersen
@emilyyount Emily Yount – Milwaukee Journal Sentinal
@eschow Emily Chow – Washington Post
@eulken Eric Ulken – Seattle Times
@fivethirtyeight Nate Silver – New York Times
@gabrieldance Gabriel Dance – Guardian
@greglinch Greg Linch – Washington Post
@GuardianData Guardian Data
@HacksHackers Hacks/Hackers
@hbillings Heather Billings – Chicago Tribune
@ireneros Irene Ros – Bocoup
@jacqui Jacqui Maher – New York Times
@JaimiDowdell Jaimi Dowdell – IRE
@JamesGleick James Gleick – author
@jasonbartz Jason Bartz – Washington Post
@jenniferamur Jennifer Amur – Milwaukee Journal Sentinal
@jenny8lee Jennifer 8. Lee – Hacks/Hackers
@jeremybmerrill Jeremy B. Merrill – ProPublica
@jeremybowers Jeremy Bowers – NPR
@jkeefe John Keefe – WNYC
@JoeGermuska Joe Germuska – Chicago Tribune
@jonathanstray Jonathan Stray – Overview Project
@jsguntzel Jeff Severns Guntzel – American Public Media
@kaeti Kaeti – MinnPost
@katdowns Kat Downs – Washington Post
@KevinQ Kevin Quealy – New York Times
@kissane Erin Kissane – Contents
@kleinmatic Scott Klein – ProPublica
@knowtheory Ted Han – Document Cloud
@kzhu91 Katie Zhu – Northwestern University
@laurenrabaino Lauren Rabaino – Seattle Times
@lenagroeger Lena Groeger – ProPublica
@lulupinney Lulu Pinney
@MacDiva Chrys Wu – Hacks/Hackers
@macloo Mindy McAdams – University of Florida
@martinstabe Martin Stabe – Financial Times
@MattDickinson44 Matthew Dickinson
@mattwaite Matt Waite – University of Nebraska-Lincoln
@mattwynn Matt Wynn – Omaha World-Herald
@MeghanHoyer Meghan Hoyer – The Virginian-Pilot
@mericson Matthew Ericson – New York Times
@michelleminkoff Michelle Minkoff – Associated Press
@mikejcorey Michael Corey – Center for Investigative Reporting
@mirandamulligan Miranda Mulligan – Knight News Lab
@mndatamine MaryJo Webster – Pioneer Press
@mtigas Mike Tigas – The Spokesman-Review
@nbrgraphs Naomi B Robbins – Forbes
@ndiakopoulos Nick Diakopoulos – researcher
@NewsDevChix Peggy Bustamante – Newsday ‏
@NiemanLabFuego Fuego – NiemanLab
@nikkiusher Nikki Usher – George Washington University
@onyxfish Christopher Groskopf – NPR
@palewire Ben Welsh – L.A. Times
@pilhofer Aron Pilhofer – New York Times
@rdmurphy Ryan Murphy – Texas Tribune
@ryanmark Ryan Mark – Chicago Tribune
@ryanpitts Ryan Pitts – Spokesman-Review
@sbhatt Sanjay Bhatt – Seattle Times
@schwanksta Ken Schwencke – L.A. Times
@SethCLewis Seth C. Lewis – University of Minnesota
@shancarter Shan Carter – New York Times
@sisiwei Sisi Wei – Washington Post
@smfrogers Simon Rogers – Guardian
@stamen Stamen Design
@stiles Matt Stiles – NPR
@tasneemraja Tasneem Raja – Mother Jones
@thejefflarson Jeff Larson – ProPublica
@tmellnik Ted Mellnik – Washington Post
@toddlindeman Todd Lindeman – Washington Post
@ultracasual Tom Meagher – Digital First Media
@wilsonandrews Wilson Andrews – Washington Post
@wjchat wjchat – Web journalist chat
@zstumgoren Serdar Tumgoren – Washington Post
@zzolo Alan Palazzolo – MinnPost

It’s interesting that you compare these software processes to building wrist watches. As a journalism and computer science double major, I am always looking to apply techniques from one area of study to the other.

There has been a lot of talk recently about how the web has changed journalism, and one huge piece of that is we no longer have one deadline at the end of the day. As is frequently quoted (and was spoken by someone unknown), “The new deadline is now.” As journalists, we are always publishing news, whether it’s through our organization’s web site or through our own personal Tweets.

We’re beginning to take a more iterative approach to writing news stories. When news breaks, we know little about it, but we piece together as much as possible in a short time frame, and we cobble together a short article. As more facts are revealed, the same story is altered. Sometimes the new information supplements what we already knew, but often these new developments completely alter what’s important in the story, generally prompting a much more thorough rewrite. Paragraphs are added, removed and rearranged throughout the process.

Since we still live in a print-focused world, this process typically ends at the end of the night, when the print deadline hits. It’s a bit like time-boxing: The reporters turn in as much as they can by the hard deadline, prioritizing the most important information in the story.

The problems that arise in software development are a lot like those of other, seemingly unrelated fields. It’s fun to apply the same techniques and see what happens.