Here¹ is what we’ll get working by the end of this walk-through (mouse-over to see load at a particular time):

Some thoughts on abstractions

Good abstractions are incredibly powerful. Bad abstractions are a massive time sink, and ultimately result in less effective code. But it’s hard, if not impossible, to really know is which before you really get into the weeds. I’ve recently started biasing away from heavy-handed abstractions, reaching instead for libraries that keep me closest to the underlying technology being used. These are frequently considerably more flexible, and relatedly, they don’t try to keep me from learning about what’s really going on, which provides benefits independent of the specific library or framework I choose to ultimately employ.

A good example of this kind of library is d3. It has frequently been noted that d3 is extremely verbose if all you’re trying to do is throw up a simple bar chart or scatter plot, but as your needs multiply and the complexity mounts, d3 scales very wells, which is to say that you don’t run into some complexity wall beyond which forcing the library to do what you want becomes the exponentially more difficult.

For this reason, I decided to use d3 when trying to put together some basic plots for energy data that I was looking at, and had hoped to turn into a standalone webapp. We’ll start with a simple line chart, and then add some interactivity at the end. As I mentioned above, the simple thing can seem daunting, but once you wrap your head around it, it’s incredible how easy it is to scale it to more complicated use-cases.

The data that we’re plotting comes from iso-ne which exposes both realtime and historical load data. The y-axis here is in Watts, but we’ll configure that further on to show more relevant scales (GW).

Baby’s first plot

Let’s start with a no-frills line-plot:

(As an aside, I’m using jekyll includes on the same .js file both in the <script/> tag that I pull in to render the chart, and the highlighted code below, so you/I can be sure they’re really the same thing).

We’ll go over this line by line, but the full javascript snippet for this graph is:

function basicChart(data) {
    // Declare the chart dimensions and margins.
    var width = 928;
    var height = 500;
    var marginTop = 20;
    var marginRight = 30;
    var marginBottom = 30;
    var marginLeft = 80;

    var x = d3.scaleUtc(d3.extent(data, function (d) { return d.timestamp; }), [marginLeft, width - marginRight]);

    var minY = d3.min(data, function (d) { return d.load; });
    var maxY = d3.max(data, function (d) { return d.load; });
    var y = d3.scaleLinear([minY - .2 * (maxY - minY), maxY + .2 * (maxY - minY)], [height - marginBottom, marginTop]);

    // Declare the line generator.
    var line = d3.line()
        .x(function (d) { return x(d.timestamp); })
        .y(function (d) { return y(d.load); });

    // Create the SVG container.
    var svg = d3.select("#my-chart-basic")
        .attr("width", width)
        .attr("height", height)
        .attr("viewBox", [0, 0, width, height])
        .attr("style", "max-width: 100%; height: auto; height: intrinsic;");
    svg.append("g")
        .attr("transform", "translate(0,".concat(height - marginBottom, ")"))
        .call(d3.axisBottom(x).ticks(width / 80).tickSizeOuter(0));
    svg.append("g")
        .attr("transform", "translate(".concat(marginLeft, ",0)"))
        .call(d3.axisLeft(y).ticks(height / 40));
    svg.append("path")
        .attr("fill", "none")
        .attr("stroke", "steelblue")
        .attr("stroke-width", 1.5)
        .attr("d", line(data))
}

Ignorning the chart dimensions, the first interesting thing here is the declaration of our scales.

var x = d3.scaleUtc(d3.extent(data, function (d) { return d.timestamp; }), [marginLeft, width - marginRight]);

var minY = d3.min(data, function (d) { return d.load; });
var maxY = d3.max(data, function (d) { return d.load; });
var y = d3.scaleLinear([minY - .2 * (maxY - minY), maxY + .2 * (maxY - minY)], [height - marginBottom, marginTop]);

Scales in d3 are incredibly simple, and do exactly what you would expect. They take two ranges (a, b) and (c, d), and give you back a function that takes some value between a and b and projects it into the range between c and d. So if you’re trying to “lift” (0, 10) into (0, 100) and you get a value of 8, you’ll want to convert that into 80, which is exactly what it does.

scaleUtc is simply a version of this function that works for timestamps.

The next thing we have is the line generator:

var line = d3.line()
    .x(function (d) { return x(d.timestamp); })
    .y(function (d) { return y(d.load); });

This helps us create the path element, which will render the curve showing load over time. This generator is just a helper function, one that we could easily build ourself with a bit of time, and it depends on those two scales we built before. We’ll eventually pass our data into this line function that gets returned, and for each data point it will figure out the corresponding x and y values using the functions we passed in here, and turn that whole thing into a path string that our browser will render using builtin SVG rendering capabilities.

The next part is where we actually grab our <svg/> element (that we’ve already added to the page, though we could add it using d3 directly if we wanted to), and append the three building blocks for our graph: the x-axis, the y-axis, and our load curve.

var svg = d3.select("#my-chart-basic")
    .attr("width", width)
    .attr("height", height)
    .attr("viewBox", [0, 0, width, height])
    .attr("style", "max-width: 100%; height: auto; height: intrinsic;");
svg.append("g")
    .attr("transform", "translate(0,".concat(height - marginBottom, ")"))
    .call(d3.axisBottom(x).ticks(width / 80).tickSizeOuter(0));
svg.append("g")
    .attr("transform", "translate(".concat(marginLeft, ",0)"))
    .call(d3.axisLeft(y).ticks(height / 40));
svg.append("path")
    .attr("fill", "none")
    .attr("stroke", "steelblue")
    .attr("stroke-width", 1.5)
    .attr("d", line(data))

First we select the <svg/> element, and this gives us back a d3-selection, which I’ve had a slightly hard time wrapping my head around. Specifically, this post, by the one of the original developers of d3, is helpful in understanding a bit about what’s going on here in the more complicated cases. For us, we’re just appending 3 elements directly.

For the axes, we depend on d3 axis functionality that takes in a g selection and adds the axis line and tick marks, based on the scales we created before (the .call(d3.axisBottom(x)) is a bit confusing, basically the selection returned from svg.append("g") exposes this .call() function which passes the created <g/> element into the d3.axisBottom(x) function. So it’s effectively the same as d3.axisBottom(x)(svg.append("g"))).

Then for the path showing the actual load curve, we use the line generator we created above. And that’s it! D3 has now added these elements into our <svg> container, and we have a graph.

This is of course significantly more complicated than wrapper libraries like Observable’s Plot, which allow you to do this all in one line all in one line (see the demo here):

Plot.lineY(aapl, {x: "Date", y: "Close"}).plot({y: {grid: true}})

Grid lines ftw

To make this a bit prettier, let’s start by scaling down those massive numbers into the 0-1000 range.

var wattPrefixes = ["", "k", "M", "G", "T", "P"];
var getMagnitude = function (min) {
    var wattages = wattPrefixes.map(function (_, idx) { return Math.pow(1000, idx); });
    var idx = wattages.findIndex(function (x) { return x > min; }) - 1;
    return [wattages[idx], wattPrefixes[idx] + "W"];
};

Now after passing in the min value of our dataset, we get [10 ^ 9, "GW"] which we’ll use to scale down the data values, and label the axes accordingly.

Let’s also update our y-axis to be a bit prettier, and add grid lines for more easily scanning left to right.

svg.append("g")
    .attr("transform", "translate(".concat(marginLeft, ",0)"))
    .call(d3.axisLeft(y).ticks(height / 40))
	// remove the domain line entirely
    .call(function (g) { return g.select(".domain").remove(); })
	// clone each tick line and then extend it all the way across
	// the graph with opacity 0.1
    .call(function (g) { return g.selectAll(".tick line").clone()
      .attr("x2", width - marginLeft - marginRight)
      .attr("stroke-opacity", 0.1);
    })
	// add some text at the top of the y-axis indicating the units
    .call(function (g) { return g.append("text")
      .attr("x", -marginLeft)
      .attr("y", 10)
      .attr("fill", "currentColor")
      .attr("text-anchor", "start")
      .text("Load (".concat(magnitudeString, ")"));
    });

Almost there:

What’s a plot without a good mouse-over

The last thing to do is add some interactivity, so a user can mouse over the graph and view the actual load at a particular time.

First we add the svg group element and set up the children elements that will show the text, along with some basic formatting options.

var dataViewLine = svg.append("g")
    .attr("id", "data-view-line");
dataViewLine
    .append("text")
    .attr("class", "date")
    .attr("fill", "currentColor")
    .attr("opacity", .6)
    .attr("text-anchor", "start");
dataViewLine
    .append("text")
    .attr("class", "load")
    .attr("fill", "currentColor")
    .attr("opacity", .8)
    .attr("text-anchor", "start");
dataViewLine
    .append("line")
    .attr("y1", height - marginBottom)
    .attr("stroke", "currentColor")
    .attr("stroke-width", .2)
    .attr("stroke-opacity", 0.0)
    .attr("y2", 0);

Then whenever the user mouses over the chart, we’ll update those elements with the corresponding data

var updateDataViewLine = function (xValue, _a) {
    // unpack args
    var data = _a.data, marginLeft = _a.marginLeft, width = _a.width, marginRight = _a.marginRight, svg = _a.svg, y = _a.y, magnitudeString = _a.magnitudeString;
	// figure out what datum this xValue is actually referring to
    var datum = data[Math.floor(data.length * (xValue - marginLeft) / (width - marginLeft - marginRight))];
    if (datum === undefined) {
        return;
    }
	// grab the data-view-line element
    var g = svg.select("#data-view-line").attr("opacity", 1.0);
    var options = { // set some time display options
        weekday: "long",
        year: "numeric",
        month: "long",
        day: "numeric",
        hour: "2-digit",
        minute: "2-digit",
        timeZoneName: "short",
    };
	// update the <text/> elements with the relevant information
    g
        .select(".date")
        .attr("x", xValue)
        .attr("y", y(datum.load))
        .text("" + new Date(datum.timestamp).toLocaleDateString("en-US", options));
    g
        .select(".load")
        .attr("x", xValue)
        .attr("y", y(datum.load) + 12)
        .text("".concat(datum.load.toFixed(2), " ").concat(magnitudeString));
    g
        .select("line")
        .attr("x1", xValue)
        .attr("x2", xValue)
        .attr("stroke-opacity", 1.0);
}

And then to configure this on our container <svg> element:

svg
    .on("mousemove", function (e) { return updateDataViewLine(d3.pointer(e)[0], context); })
    .on("mouseover", function (e) { return updateDataViewLine(d3.pointer(e)[0], context); })
    .on("mouseout", function () {
    svg.select("#data-view-line")
        .attr("opacity", 0.0);
});

(d3.pointer gives us the relevant coordinates for the event within our graph).

Note that we’re not magic here to make this work. We’re getting some x value from the mouse event, translating it into our graph coordinates, and then figuring out which datum from our array of load data this actually corresponds to. That logic is all here:

var datum = data[Math.floor(data.length * (xValue - marginLeft) / (width - marginLeft - marginRight))];

This seems a bit messy at first glance, but it’s also incredibly straightforward, and that trade-off is in this case I think very much worth it. So with all that said and done, we finally have our interactive graph of energy data.

I might be using the term interactive pretty generously. ↩