Although I wasn’t here in Seattle for the 1999 globalization protests, or the 2012 vandalism, it’s hard to miss the city’s regular May Day “festivities.” About a week ago, the digital team started thinking about how to track and collect information from reporters as they followed activities in the city. A map is a natural way to collect this data, but I didn’t want to manually manage geotagging and attaching media. Was there some way to pull this information directly from social networks?

My first stop was NPM to see if there was a module for Twitter streams, which (as it happens), there was. Hooking into the streaming API gave us real-time updates with a variety of filters available, including geotagged location and text search, while using the regular REST API let us backfill for tweets we missed and look up users for more information.

We’re wary about relying on user content anyway, but it quickly became clear that most people don’t enable location services on Twitter, and even fewer use the “precise” option, which must be re-enabled for each tweet. Clearly, we couldn’t just rely on the general population to feed our map. Our new plan was to teach reporters and photographers how to tag their locations, and then follow a real-time stream of whitelisted accounts.

I quickly put together a map-and-stream prototype to demonstrate that the concept was feasible, and during our planning meeting I fired off a quick post from a temporary test account. Luckily, the process worked, and immediately a dot appeared on the map with this picture of our charismatic digital and photo staffers:

One advantage we had in quickly assembling this map was our leaflet-map component, which assembles a map from a domain-specific set of custom elements. Because it handles setting up the Leaflet instance and populating it from markup, I could output the initial state of the map straight from our HTML templates (including zoom, position, tile layers, and map markers) instead of having to write error-prone initialization code. The same JSON file that feeds that template is also uploaded to S3, and client-side code checks it every 15 seconds for updates, so readers don’t have to refresh the page. In fact, the interactive team watched the march’s progress all day, just by leaving the page open.

It was important for readers to be able to instantly see where the action currently was, and how the marches moved across the city. We took advantage of the fact that we run JavaScript on both the client and server to share the aging algorithm between the two, and then set a class on map markers with the result. Newer tweets are more saturated, and those with video got a different color to differentiate them from text and images.

If you’re interested in putting together your own Twitter map, feel free to check out the source code for the map on GitHub. It’s based on our news app template, but with an additional index.js script that connects to Twitter and re-runs the build whenever there are updates. We also use the sqlite3 module to store tweets between server restarts, which is something we haven’t done before. Integrating a database works well, but locks us to a single machine for deployments.

While I’m mostly happy with our visualization, I think there are many improvements we can make, and a project like this is a great way of learning where we need to improve the leaflet-map map element and other in-house libraries. For example, I want to be able to link tweets in the stream with dots on the map, which requires the ability to set attributes on the markers beyond class names. I’d also like to be able to filter both dots and stream by reporter and time, so that readers can “replay” events for a better sense for their movement over time.

Another limitation that we didn’t have time to work around is the problem of Twitter video, which the service keeps deeply inside of its walled garden. Although video has been embeddable on the service for most of this year, it’s still not exposed via the API (it sends down a thumbnail picture instead, and there’s no indication that it’s different from any other photo). We worked around this problem by manually tagging tweeets with #video, coloring those dots differently, and then linking the thumbnail directly to the tweet, but it’s an inelegant solution at best.

Still, I’m very pleased with how the map turned out, especially given that we put it together in only about two days — that’s coding under pressure! More importantly, it wouldn’t have been possible to do, or nearly as amazing to watch, without the fantastic work of the reporters and photographers who fed it with their coverage of the event, risking police flashbangs, teargas, and pepper spray to do so. We’re incredibly thankful to @seattlesullivan, @pgcornwell, @seaclarridge, @stevemiletich, @stimesmcarter, @katherinelong, @evanbush, @eriklacitis, @jbroomseattle, @mikelindblom, @jim_brunner, @deleon_times, @gina_cole_, @sjgtimes, @sringman, @deanrutz, @bettinahansen, @lindseywasson, @laurenfrohne, @kgcotterill, @corinnechin, @gawlowski, and @colindilt for all their help.