CategoryJavaScript

How we adapted the LA Times map maker

When news breaks, The Seattle Times faces a common problem: we want visual elements for the story, but photographers may not be on the scene yet. For this, we’ve often turned to maps from the graphics department, but these are labor-intensive to build, even with our fantastic crew of news artists. Services to generate maps are expensive. What’s a small, penny-pinching newsroom to do?

My first attempt at solving this problem was a simple Leaflet-based app. The styling wasn’t great, and due to browser security restrictions, people still had to manually screenshot the window in order to grab an image. It was, in short, a total hack. That’s why I was thrilled to discover the work that the LA Times Data Desk has been doing with their Web Map Maker project. Using OpenStreetMap data and vector tiles from MapZen, their tool is capable of rendering directly to a canvas, and then saving that image–no screenshot required!

I forked the project pretty much right away, and started messing around with it. Eventually, an evening in Sublime turned into a substantial rewrite, with two main goals: to match Seattle Times code style, and to simplify it by taking advantage of new browser features. In this post, I’d like to walk through some of the changes that we made–not as a criticism of the LA Times’ work, which I’m still in awe of, but as a way to explore how modern JavaScript can make a real-world application simpler and easier to maintain.

async/await

The first thing I noticed about the web map maker was the downloadIMG() function, which does the actual image capture. At more than 100 lines, it’s a monster, but a necessary one: it combines canvas rendering for the base map, html2canvas to grab popups and other map elements, and a chunk of custom code to draw any SVG elements loaded from GeoJSON. Web maps are complicated!

Compounding this problem is that, like a lot of legacy JavaScript, the code is callback-heavy. downloadIMG() becomes more and more indented as it moves through various stages, which is hard to maintain. Untangling this rendering process made a lot of sense as a starting point for the refactor, and using async/await was a natural method for taming those wild callbacks.

The async and await keywords are new to ES2017, but you can use them in evergreen browsers like Chrome and Firefox now. They’re syntactic sugar for the then() method of JavaScript Promises: inside of a function marked with async, you can “pause” the function using await instead of passing a callback to then(). For example, this:

var asyncOperation = functionThatReturnsAPromise();
asyncOperation.then(function(result) {
  return doSomeWork(result);
}).then(function(nextResult) {
  return doMoreAsyncWork(nextResult);
});

can be simplified into this:

var firstResult = await functionThatReturnsAPromise();
var secondResult = await doSomeWork(firstResult);
await doMoreAsyncWork(secondResult);

At each await, the browser converts your function into a callback for you, even in the middle of an expression or a variable assignment. The resulting code no longer requires all the function boilerplate and extra indentation. It can be “read” as synchronous, even though in reality it might take significant time to execute.

Breaking downloadIMG() into several promise-based functions made it much easier to understand and debug, since skipping a step became as simple as commenting out an await-ed function call. The new code has actually gotten a little longer after being broken up into sub-tasks, but the readability is much higher now. I was also able to move it out into its own module, which strengthens the distinction between logic and UI in the app structure.

No jQuery

jQuery is great, but at the Times we have a policy of avoiding it unless we need at least three of its hard-to-shim features, like JSONP requests or event delegation. Modern browser APIs have learned a lot from jQuery, and integrated many of its features, to the point where you might not need its help.

The original map maker code used jQuery extensively, but most of the cleanup was pretty straightforward:

  • perform DOM queries with a call to document.querySelectorAll(), aliased to $
  • use classList instead of addClass() and removeClass() calls
  • addEventListener() instead of on()

A common pattern of jQuery is that it acts on all elements in a selected query, whether there’s one or more (or even zero). ES2015’s arrow functions aren’t quite that convenient, but they do provide a close analog:

// jQuery version
$(".toggle-element").addClass("enabled");

// qsa() returns an array from document.querySelectorAll
qsa(".toggle-element").forEach(el => el.classList.add("enabled"));

It’s a little more clumsy, but it saves us the cost of loading 32KB of code for jQuery, which contributes to faster page load. jQueryUI adds another 55KB, most of which is unused–the project only needs the resize functionality for the map’s drag handle. I write a small vanilla JS module to do resizing instead, dispatching a custom DOM event whenever the element’s dimensions changed so that we could continue listening in the main module for changes.

Eliminating ~90KB of code may not seem like a lot, but on lower-powered devices, that can shave ~90ms off the page load just from the cost of code parsing and module execution. It also means we no longer load the CSS required for jQueryUI (which, ironically, the original app did not use anywhere). And speaking of CSS…

Flexbox instead of Bootstrap

This is another technique that’s becoming conventional wisdom as browsers improve: you might not need Bootstrap! The Flexbox specification for CSS lets us define a page in terms of vertical or horizontal “flex containers,” and force the children of those containers to stretch or align on either axis (finally, easier vertical centering is here). The original map maker used Bootstrap for some button group styles, but otherwise didn’t really rely on it. Swapping Flexbox in for positioning, and styling those buttons manually, turned out to be a pretty easy fix.

Destructuring and for/of loops

Finally, although it’s not a big change, I did take advantage of destructuring assignment throughout the app. For example, when pulling the width and height out of the resizer’s event:

var { width, height } = event.detail;

Or even when importing modules from my async-based loader:

var [ qsa, resizer, render ] = await install.batch("qsa", "resizer", "render");

And when combined with the new value-oriented loops and iterables in ES2015, it eliminates some boilerplate from checking the current dimensions against existing size presets:

for (var [w, h] of Object.values(sizePresets)) {
  if (w == width && h == height) custom = false;
}

This kind of unpacking behavior is common in Python or PHP, so it’s a relief to finally have it in JavaScript. It reduces the assignment code you need to write, and makes it clearer which properties you’re going to use from a given object. In some cases, it can also make tight loops faster, since local variables are quicker than properties accessed through the dot operator.

Final thoughts

Internal tools like this are a great chance to play around with new browser features, because we can set stronger minimum requirements for users. However, you can also publish code that uses many of these new JavaScript toys if you’re willing to set up a transpiler like Babel in your build process. We’ve integrated transpilation as a part of our standard news app template at the Times, and it has made a lot of our code much more pleasant to read and write. I highly recommend giving it a shot.

I’d like to close by again noting how incredibly grateful I am that the LA Times decided to open-source their tools, and that LAT map guru Jon Schleuss was so gracious about sharing with us. We’re actually still working on the project styles, and hoping to use it in production soon. You can check out the repo and watch our progress here.

Unfortunately, it’s likely that I’ve refactored so much that it’ll be difficult to pull changes back into their master branch. I hope that by spelling out the changes in this post, it’ll be easier to figure out which parts are most useful, and which ones are just me showing off. Projects like this really drive home just how important open-source development can be for the news nerd community, since we’re much stronger as a community than we are as individual newsrooms.

Roaring to life: How we built our Elwha watershed package

When Lynda Mapes, our environment reporter, came to us with the story of how the Elwha watershed had been recovering after the world’s largest dam removal, we knew we wanted to present it in a way that would make a splash. The final result (GitHub repo) drew heavily on talents from all over the newsroom, ranging from the graphics team’s hand-drawn art to Lynda’s beautiful prose. On the digital side, there were three interesting JavaScript components that deserve more explanation: the watercolor animations, scroll-triggered effects, and FLIP transitions.

You otter be in pictures

Black and white river otter sketch

Black and white base layer

Full color otter sketch

Color “paint” layer

We knew going into this project that we would want to feature the beautiful hand-drawn images of various critters in the Elwha watershed. The final technique is an adaptation of the “lead dust” effect that we used in our Loaded with Lead investigative report. Instead of a single texture that’s used to “reveal” hidden text, we split the sketches into two layers: one base image that only contained the black channel for shading, and another full-color image that gets painted on top. By using intentionally-imperfect circles as our “brushes,” and expanding these circles from randomized positions, the resulting “watercolor” effect helps add a simple splash of life to the static images, without feeling gratuitous or jarring.

Ultimately, what makes this effect possible (and performant) on a range of devices is the canvas createPattern() method, which takes an image and generates a fill pattern object. Most of the time, this is used with smaller images to add texture to a filled path, so that you don’t need to draw and clip the pattern by hand. But in this case, the pattern is the same size as the canvas itself, meaning that we can use this to copy pieces of one image into the canvas in irregular patches–perfect for our watercolor wash. (source code)

Scroll me maybe

Another key part of the presentation is that many animations trigger as they become visible, rewarding readers for scrolling down the page. This is a pretty typical part of news interactive UI these days, but it’s worth looking at how to do it without causing jank.

Our secret weapon for many scroll-triggered effects at the Seattle Times is getBoundingClientRect(), one of the many useful DOM methods that (surprisingly) originated in Internet Explorer. When called on any element, it returns an object with the coordinates for that element relative to the viewport. In other words, it tells you where the object is displayed right now, without having to walk up the DOM for offset values or check the current scroll position. This leads to easy visibility tests like this one:

var bounds = element.getBoundingClientRect();
if (bounds.top < window.innerHeight && bounds.bottom > 0) {
  //element is onscreen in some form
}

getBoundingClientRect() is fast and easy to use for a variety of purposes, and it’s well-supported everywhere. While browsers continue to work to make scrolling smoother, it’s still important to do as little work on scroll listeners as possible, especially when the page may contain many of them (as the Elwha package does). We created a unified scroll notification queue for this project to try to keep the load down. Which brings us to our last animation feature…

FLIP and forget it

I’m a big fan of Paul Lewis’ FLIP animation technique, which leverages the browser to perform smooth animations with as little JavaScript as possible. It’s worth reading the article for more details, but the acronym spells out the basic steps: get the first position, move to the last position, invert the difference through the use of transforms, and then play by removing the offsets.

Elwha feature section

One of our animated feature sections. The animals on the right swap into the main section with a smooth expand effect when clicked.

The nice thing about FLIP is that it’s not only buttery-smooth, but it also adapts well to various lower-level page changes. In this case, when a user clicks on one of the sidebar items to swap between featured animals, we actually move the entire block between two different container elements for styling purposes. But because FLIP works entirely via on-screen coordinates and transforms, none of that  matters. We only have to know the first and last positions to create the transform, and those are easy to get via our old pal getBoundingClientRect().

Although the code for FLIP isn’t hard to write, it comes up often enough (see also: EB-5 visas, teacher pay, and modern dating) that we’ve added a small module to our news app template to make it even easier. The function takes two arguments: an element that you want to animate, and a callback that’s executed between measurements. Just make whatever changes you want during the callback, and the module will figure out the difference and apply transforms to blend between the two. Feel free to take and adapt for your own purposes!

Introducing our news app template

Hi, I’m Thomas Wilburn, newsroom web developer here at The Seattle Times. I work with editors and reporters to tell stories on the web, ranging from data visualizations to custom news applications. One of my most important tools for putting out great projects under deadline pressure is our news app template.

Digital storytelling is not new at The Seattle Times — you only have to look to Sea Change or our election guides to see that — but it hasn’t had a consistent process for development. Some of our news apps were built in Django, some in WordPress, and others in notepad.exe, depending on the staff assigned and their mood at the time. When I joined the newsroom earlier this year, one of my goals was to create a standard platform for digital projects, generating static files for ease of maintenance and low-stress hosting.

The result is a scaffolding system built on top of Grunt and NodeJS for producing news applications with the absolute minimum of friction (editorial or technical). In addition to populating a project with boilerplate HTML, the project initialization process also sets up the following helpful features:

  • A local development server, with live reload integration
  • Lo-dash templates, with some useful built-in helper functions
  • LESS and JavaScript compilation via AMD modules
  • A “watch” server to run build tasks automatically whenever files change
  • Bower pre-configured to install client-side libraries in a standard location
  • The ability to pull data from CSV, JSON, or Google Docs
  • One-command publication to Amazon S3

In many ways, this list is similar to scaffolding solutions from other organizations, including the NPR app template and the Chicago Tribune’s Tarbell. However, being built on NodeJS, the Seattle Times template is a bit easier to set up, and runs on more diverse software (namely, Windows). As a result, it’s been easy to get our web producers working on the same stack that we use for our big projects.

Our experiences using this app scaffolding have been positive so far. Using this scaffolding, we can be up and running on a new project in minutes, and the common structure means that it’s easy to reuse components from one app to another. Fast deployment makes our development faster, and being able to pull from Google Docs makes it easier to bring in editors and reporters. If they can use a spreadsheet, they can edit our interactives. We’ve used it to power many of our online features this year, including “Where the Bidding Wars Are?” and our Oso Landslide timeline. It even runs our Seahawks fan map!

As big proponents of open-source software, our team believes this kind of slick development experience is just too cool to keep to ourselves. So we’ve made our scaffolding available on GitHub under a GPL license. There are a few Seattle Times-specific bits you’ll need to adapt if you use it for yourself, such as our ad and tracking code. But other than that, I think it could be useful for anyone building static sites — inside or outside of a newspaper. If you build something with it, we’d love to hear about it!