This folio package was a two-page spread in the paper focusing on poor prospects for recent grads. It wouldn’t translate well online: not only would the text appear in a linear way, but you’d lose the scanability from the blurbs.

I built this interactive using gRaphael… for some reason. Really, it would have been easier to create static images since the interactivity was low. But it was good practice and an exercise in turning around an interactive very quickly.

See the full version.

I’m honoured to have one of my interactives nominated for the first annual Data Journalism Awards. The international competition is supported by Google and run by the Global Editors Network. It’s also the first of its kind in the world.

The Globe submitted a handful of pieces and the Sunshine List interactive was nominated for international/national data application. The sortable, searchable table took a couple weeks of scraping, refining and developing to get in working order. I also found a way to search the previous year’s records to show any change in income or benefits, adding an interesting layer to the data.

There are some fantastic entires from The Guardian, The Australian, the BBC, The Toronto Star and more. See all the nominees.

This interactive lets you explore mucho data behind Vancouver’s high schools, from capacity to graduation rates.

It was created using Raphael, a Javascript library that creates vector images, so it’s great for maps. I also used gRaphael to create an animate some pie charts and bar charts.

My favourite part, though, came from mapping the location of all the students. It took a lot of work — resizing the dots, scaling and positioning them to match the native data. But the final result is pretty cool.

See the full interactive.

I’ve started a new collective, kind of like the Open House Arts Collective of yore, but this time about data journalism. It’s called Ad Hoc Data. It’s a chance to work with like-minded nerds interested in exploring the tech and story-telling that comes from data.

Our first project, a federal budget calculator, took quite a bit of time and offered some interesting results. Try it yourself.

I’m excited to see where it goes next. It’s a chance for me to experiment and create with more freedom than usually afforded to me. And a chance to meet other like-minded folks interested in creating, creating and creating.

The Ontario government released their annual Sunshine List on March 24, detailing public sector employees earning more than $100,000 per year. I created a table so readers can explore the list in more detail, letting you can search, sort and filter by name, salary and more.

This list is published each year on the Government’s website in a way that’s hard to search, impossible to sort and difficult to navigate. The Globe wanted to pull the data from this year’s list and publish it in a more usable way, as a tool for our reporters and our readers.

Here’s a little background on how I made the tool.

I started by building a scraper, a program that trolls web pages for content and saves it in a more sophisticated way than copy-paste. Using a coding language called Python, I built a universal scraper that could pull all the data back to 1997 – the first year it was released.

I cleaned this data using Google Refine, converting encoded HTML characters and renaming some categories. The next challenge was cutting the data down as much as possible. While there were only a few thousand records back in the 1990s, other years had as many as 79,000 records, making the file sizes very large. While Chrome and Firefox could handle it well, Internet Explorer chugged slowly with each new megabyte I pushed its way.

So I divided the master file into several chunks by category and opted to not use a JSON array, since the keys added unnecessary kilobytes to the file. A standard JavaScript array was used instead.

Usually when you’re dealing with a big dataset, programmers use server-side language like PHP to query the data. But we were wary of doing this because the technical and administrative overhead seemed insurmountable within our timeframe. So we tried exploring browser-side options and settled on SlickGrid, an open-source JavaScript plug-in that handles massive amounts of data very well. The plug-in had to be customized to handle some extra functionality: currency sorting, historical comparisons and a universal search box.

Since I had data from the 2010 release, I added a feature to let readers compare increases or decreases. In an earlier version of the table, I also included the employer name and position with this pop-up. But I had to cut it late in development because it nearly doubled the size of the 2010 dataset.

The final tool is very simple to use and, admittedly, not very flashy. But it lets readers dig a little deeper into the list, find notable people or search specific jobs.

View the full interactive.

This map takes data from the Toronto Traffic Safety Unit and plots over 2,000 points measuring traffic volume. I’ve also added a unique view showing just the top 100, where the radius of each circle corresponds to the number of cars passing through that point. Finally, we used on qualitative data to plot the ten most congested points in the city.

Each uses a different technique: the first uses standard Fusion Tables plotting; the second uses a custom Fusion Tables query and several calculations to plot circle objects onto the map; the third makes another query and uses the latitude and longitude as coordinates to plot custom icons. Overall this took about a week and accompanies a great story on traffic congestion.

My latest map plots the various hazards that threaten the Northern Gateway pipeline — from underground earthquakes to landslides to protected animal populations. The pipeline ends at Kitimat, B.C., a small coastal port on the west Coast. From there, oil’s transferred onto tankers and tugged through the maze of the Douglas Channel. There are three exit opens but only one approved anchor point. It’s an interesting look at the tremendous difficulty in getting oil out of Canada.

View the full interactive

After scraping the content for all the Drummond reports, I wanted to show them in a simple way. Inspired by What The F#*! Should I Make For Dinner, I created this simple site that shows a random recommendation with option to tweet the ones you like.

The site works by pulling in all the recommendations in a JSON file, then creating a random number and pulling that record. I also added a query option, so if you add “?37″ to the end, it’ll pull the 37th recommendation. This is helpful for tweeting and recalling an interesting recommendation.

Visit the site.

The Drummond Report offered mucho insight into Ontario’s future. But the recommendations themselves were buried in 300 pages of background and chitter. What if you just wanted the recommendations? You could browse all 20 chapters on the government website. Or, with some deft scraping, you could pull them down and throw them into a table of your own.

That’s what I did last week with the help of ScraperWiki, a super-handy website that gets you up-and-running with Python, Mechanize and other scraping libraries in no time flat.

Download the Drummond report recommendations (CSV) or visit the website for the full text.

How the scrape worked

Here’s a look at the full scrape from ScraperWiki. You can see this on the site too.

import scraperwiki           
import lxml.html
import re
from BeautifulSoup import BeautifulSoup

for x in range(1,21):
    src = 'http://www.fin.gov.on.ca/en/reformcommission/chapters/ch%d.html' % (x)
    html = scraperwiki.scrape(src) 
    soup = BeautifulSoup(html)    
    filter = soup.findAll('p')
    for y in filter:
        z = y.find(re.compile('^strong'))
        if z: 
            a = z.find(text=re.compile('^Recommendations*'))
            if a: 
                location = y.getText().split(':')[0]
                text = y.getText().split(':')[1]
                chapter = location.split('-')[0].split(' ')[1]
                recId = location.split('-')[1]
                data = {
                'rec':chapter + '-' + recId,
                'chapter':chapter,
                'recId':recId,
                'data':text
                }
                scraperwiki.sqlite.save(unique_keys=["rec"], data=data)

									

Breaking this down a bit, we first import a few necessary scripts, the most important of which (for our purposes) is BeautifulSoup, a text parser that makes it really easy to drill into HTML and find specific tables, tags and text.

Next, I’ve set up a loop that cycles from chapter 1 to chapter 20, then dynamically subs in the appropriate chapter number. This is a Python variable here: wherever %d is displayed, it’ll get the variable from the brackets at the end. In this case, %d stands for x, which is the chapter number.

    src = 'http://www.fin.gov.on.ca/en/reformcommission/chapters/ch%d.html' % (x)
									

I’ve taken this HTML and pushed it through BeautifulSoup, which gives me access to all new functions like find() and findAll(). In this case, I want just the recommendations, which are conveniently stored in <strong> tags, which are stored in paragraphs. So each recommendation looks something like:

<p><strong>Recommendation 1-1:</strong> This is the first recommendation...</p>
									

So on each page, I want to look at all the paragraphs. This translates to soup.findAll(‘p’). Now that I have them, I want to go through each one and determine if it begins with a bold (or strong) tag. So y.find(re.compile(‘^strong’)). The ‘^’ symbol is a regex symbol for “at the beginning of.” I don’t much care if there’s a strong tag somewhere else in the paragraph; if it’s not at the beginning, it’s not a recommendation.

But to be double-sure, I also want to check that strong tag for the word recommendation. So each time I find the strong tag, I also check if there’s the word “Recommendation” at the beginning of it. Thus:

       z = y.find(re.compile('^strong'))
        if z: 
            a = z.find(text=re.compile('^Recommendations*'))
									

Now I’m sure I’ve found the word “Recommendation” at the start of a <strong> tag, which is at the start of a paragraph (<p>) tag. Phew! Now, under frabcus‘ suggestion, I also want to store the chapter and recommendation number. So I’ve split the text inside the <p> tag at the first colon (returning all the text before “Recommendation 1-1:”), then split this a bit further on the “-” tag, then on the space. This gives me two values, the first being the first number and the second being the second number. For reasons unclear at the moment, it sometimes didn’t get a value here, but no bother. I can clean this up in the final version.

I’ve then pushed this all into a ScraperWiki table using their SQLite save() command. To do that, I needed a unique key for each row, which I created using the chapter and recommendation number in the text itself.

Any questions? I’m happy to help where I can.

Google has a lovely suite of tools for creating custom maps, chief among them the Maps API and Fusion Tables. They work together like brother and sister to create respectable visualizations with mucho data. Let’s dive in.

Get your data ready

Your data should be in a clean CVS or XLS table. You’ll probably have to do some work refining things first. (Hey, check out Google Refine for that!) Then spend some time doing simple graphs and interviewing the data in Excel for trends and clues.

When you have an idea of what you want to map, open up Fusion Tables.

For the purposes of this tutorial, let’s use my spreadsheet on B.C. federal prisons, available on BuzzData.

Uploading to Fusion Tables

Go to Fusion Tables (click “see my tables” from the splash screen). But DO NOT try to just upload the file to Google Docs. Instead, go to Create > Table. Choose your file, preview, upload.

Once loaded, you should see the table. A bunch of fields are listed at the top, most importantly Latitude and Longitude. Fusion Tables recognizes these fields as coordinates so you don’t have to do any work to map it.

If you had latitude and longitude but these were called “lat” and “lon” or “x” and “y,” you can tell Fusion Tables these are coordinates by going to Edit > Modify Columns and changing the Type of columns.

Since they’re already specified, go to Visualize > Map. And presto! You have a map. You can embed it at this point, but it’s really ugly and you get an even uglier InfoWindow when you click on a marker.

 Make it prettier

There are two simple tools you need to use here:

1. Configure styles: this changes the markers
2. Configure info windows: this changes the pop-up window

Clicking Save refreshes the map with the new colours. You should see at least one of them at your zoom level with a yellow marker.

Next, click configure info windows. Fusion Tables feeds some info here automatically, but it’s pretty hideous. So go to the second tab, Custom, and add some HTML. I’ve prepared this simple version:

<div class='googft-info-window' style='font-family: sans-serif'>
<h3>{Prison}</h3>
<strong>Date opened:</strong> {Date Opened}<br />
<strong>Number of inmates:</strong> {Number of inmates}<br />
<hr />
<strong>Total seizures:</strong> {total seizures}
</div>
									

Now it looks like this:

Spiffy! Now you can embed this in your site. First, go to Share in the top right and set it to Unlisted, then hit Save. Then click on Get embeddable link.

But that’s no fun. You have no legend and no controls. Plus there’s more you could do with the map, like custom icons and manual filters.

In Part 2, we’ll visualize the map using the Maps API and the FusionTablesLayer to add filters, custom icons and views.