Neil’s Blog

Interactive visualization of energy data with d3

2023-11-06T17:51:15+00:00

Here¹ is what we’ll get working by the end of this walk-through (mouse-over to see load at a particular time):

Some thoughts on abstractions

Good abstractions are incredibly powerful. Bad abstractions are a massive time sink, and ultimately result in less effective code. But it’s hard, if not impossible, to really know is which before you really get into the weeds. I’ve recently started biasing away from heavy-handed abstractions, reaching instead for libraries that keep me closest to the underlying technology being used. These are frequently considerably more flexible, and relatedly, they don’t try to keep me from learning about what’s really going on, which provides benefits independent of the specific library or framework I choose to ultimately employ.

A good example of this kind of library is d3. It has frequently been noted that d3 is extremely verbose if all you’re trying to do is throw up a simple bar chart or scatter plot, but as your needs multiply and the complexity mounts, d3 scales very wells, which is to say that you don’t run into some complexity wall beyond which forcing the library to do what you want becomes the exponentially more difficult.

For this reason, I decided to use d3 when trying to put together some basic plots for energy data that I was looking at, and had hoped to turn into a standalone webapp. We’ll start with a simple line chart, and then add some interactivity at the end. As I mentioned above, the simple thing can seem daunting, but once you wrap your head around it, it’s incredible how easy it is to scale it to more complicated use-cases.

The data that we’re plotting comes from iso-ne which exposes both realtime and historical load data. The y-axis here is in Watts, but we’ll configure that further on to show more relevant scales (GW).

Baby’s first plot

Let’s start with a no-frills line-plot:

(As an aside, I’m using jekyll includes on the same .js file both in the <script/> tag that I pull in to render the chart, and the highlighted code below, so you/I can be sure they’re really the same thing).

We’ll go over this line by line, but the full javascript snippet for this graph is:

function basicChart(data) {
    // Declare the chart dimensions and margins.
    var width = 928;
    var height = 500;
    var marginTop = 20;
    var marginRight = 30;
    var marginBottom = 30;
    var marginLeft = 80;

    var x = d3.scaleUtc(d3.extent(data, function (d) { return d.timestamp; }), [marginLeft, width - marginRight]);

    var minY = d3.min(data, function (d) { return d.load; });
    var maxY = d3.max(data, function (d) { return d.load; });
    var y = d3.scaleLinear([minY - .2 * (maxY - minY), maxY + .2 * (maxY - minY)], [height - marginBottom, marginTop]);

    // Declare the line generator.
    var line = d3.line()
        .x(function (d) { return x(d.timestamp); })
        .y(function (d) { return y(d.load); });

    // Create the SVG container.
    var svg = d3.select("#my-chart-basic")
        .attr("width", width)
        .attr("height", height)
        .attr("viewBox", [0, 0, width, height])
        .attr("style", "max-width: 100%; height: auto; height: intrinsic;");
    svg.append("g")
        .attr("transform", "translate(0,".concat(height - marginBottom, ")"))
        .call(d3.axisBottom(x).ticks(width / 80).tickSizeOuter(0));
    svg.append("g")
        .attr("transform", "translate(".concat(marginLeft, ",0)"))
        .call(d3.axisLeft(y).ticks(height / 40));
    svg.append("path")
        .attr("fill", "none")
        .attr("stroke", "steelblue")
        .attr("stroke-width", 1.5)
        .attr("d", line(data))
}

Ignorning the chart dimensions, the first interesting thing here is the declaration of our scales.

var x = d3.scaleUtc(d3.extent(data, function (d) { return d.timestamp; }), [marginLeft, width - marginRight]);

var minY = d3.min(data, function (d) { return d.load; });
var maxY = d3.max(data, function (d) { return d.load; });
var y = d3.scaleLinear([minY - .2 * (maxY - minY), maxY + .2 * (maxY - minY)], [height - marginBottom, marginTop]);

Scales in d3 are incredibly simple, and do exactly what you would expect. They take two ranges (a, b) and (c, d), and give you back a function that takes some value between a and b and projects it into the range between c and d. So if you’re trying to “lift” (0, 10) into (0, 100) and you get a value of 8, you’ll want to convert that into 80, which is exactly what it does.

scaleUtc is simply a version of this function that works for timestamps.

The next thing we have is the line generator:

var line = d3.line()
    .x(function (d) { return x(d.timestamp); })
    .y(function (d) { return y(d.load); });

This helps us create the path element, which will render the curve showing load over time. This generator is just a helper function, one that we could easily build ourself with a bit of time, and it depends on those two scales we built before. We’ll eventually pass our data into this line function that gets returned, and for each data point it will figure out the corresponding x and y values using the functions we passed in here, and turn that whole thing into a path string that our browser will render using builtin SVG rendering capabilities.

The next part is where we actually grab our <svg/> element (that we’ve already added to the page, though we could add it using d3 directly if we wanted to), and append the three building blocks for our graph: the x-axis, the y-axis, and our load curve.

var svg = d3.select("#my-chart-basic")
    .attr("width", width)
    .attr("height", height)
    .attr("viewBox", [0, 0, width, height])
    .attr("style", "max-width: 100%; height: auto; height: intrinsic;");
svg.append("g")
    .attr("transform", "translate(0,".concat(height - marginBottom, ")"))
    .call(d3.axisBottom(x).ticks(width / 80).tickSizeOuter(0));
svg.append("g")
    .attr("transform", "translate(".concat(marginLeft, ",0)"))
    .call(d3.axisLeft(y).ticks(height / 40));
svg.append("path")
    .attr("fill", "none")
    .attr("stroke", "steelblue")
    .attr("stroke-width", 1.5)
    .attr("d", line(data))

First we select the <svg/> element, and this gives us back a d3-selection, which I’ve had a slightly hard time wrapping my head around. Specifically, this post, by the one of the original developers of d3, is helpful in understanding a bit about what’s going on here in the more complicated cases. For us, we’re just appending 3 elements directly.

For the axes, we depend on d3 axis functionality that takes in a g selection and adds the axis line and tick marks, based on the scales we created before (the .call(d3.axisBottom(x)) is a bit confusing, basically the selection returned from svg.append("g") exposes this .call() function which passes the created <g/> element into the d3.axisBottom(x) function. So it’s effectively the same as d3.axisBottom(x)(svg.append("g"))).

Then for the path showing the actual load curve, we use the line generator we created above. And that’s it! D3 has now added these elements into our <svg> container, and we have a graph.

This is of course significantly more complicated than wrapper libraries like Observable’s Plot, which allow you to do this all in one line all in one line (see the demo here):

Plot.lineY(aapl, {x: "Date", y: "Close"}).plot({y: {grid: true}})

Grid lines ftw

To make this a bit prettier, let’s start by scaling down those massive numbers into the 0-1000 range.

var wattPrefixes = ["", "k", "M", "G", "T", "P"];
var getMagnitude = function (min) {
    var wattages = wattPrefixes.map(function (_, idx) { return Math.pow(1000, idx); });
    var idx = wattages.findIndex(function (x) { return x > min; }) - 1;
    return [wattages[idx], wattPrefixes[idx] + "W"];
};

Now after passing in the min value of our dataset, we get [10 ^ 9, "GW"] which we’ll use to scale down the data values, and label the axes accordingly.

Let’s also update our y-axis to be a bit prettier, and add grid lines for more easily scanning left to right.

svg.append("g")
    .attr("transform", "translate(".concat(marginLeft, ",0)"))
    .call(d3.axisLeft(y).ticks(height / 40))
	// remove the domain line entirely
    .call(function (g) { return g.select(".domain").remove(); })
	// clone each tick line and then extend it all the way across
	// the graph with opacity 0.1
    .call(function (g) { return g.selectAll(".tick line").clone()
      .attr("x2", width - marginLeft - marginRight)
      .attr("stroke-opacity", 0.1);
    })
	// add some text at the top of the y-axis indicating the units
    .call(function (g) { return g.append("text")
      .attr("x", -marginLeft)
      .attr("y", 10)
      .attr("fill", "currentColor")
      .attr("text-anchor", "start")
      .text("Load (".concat(magnitudeString, ")"));
    });

Almost there:

What’s a plot without a good mouse-over

The last thing to do is add some interactivity, so a user can mouse over the graph and view the actual load at a particular time.

First we add the svg group element and set up the children elements that will show the text, along with some basic formatting options.

var dataViewLine = svg.append("g")
    .attr("id", "data-view-line");
dataViewLine
    .append("text")
    .attr("class", "date")
    .attr("fill", "currentColor")
    .attr("opacity", .6)
    .attr("text-anchor", "start");
dataViewLine
    .append("text")
    .attr("class", "load")
    .attr("fill", "currentColor")
    .attr("opacity", .8)
    .attr("text-anchor", "start");
dataViewLine
    .append("line")
    .attr("y1", height - marginBottom)
    .attr("stroke", "currentColor")
    .attr("stroke-width", .2)
    .attr("stroke-opacity", 0.0)
    .attr("y2", 0);

Then whenever the user mouses over the chart, we’ll update those elements with the corresponding data

var updateDataViewLine = function (xValue, _a) {
    // unpack args
    var data = _a.data, marginLeft = _a.marginLeft, width = _a.width, marginRight = _a.marginRight, svg = _a.svg, y = _a.y, magnitudeString = _a.magnitudeString;
	// figure out what datum this xValue is actually referring to
    var datum = data[Math.floor(data.length * (xValue - marginLeft) / (width - marginLeft - marginRight))];
    if (datum === undefined) {
        return;
    }
	// grab the data-view-line element
    var g = svg.select("#data-view-line").attr("opacity", 1.0);
    var options = { // set some time display options
        weekday: "long",
        year: "numeric",
        month: "long",
        day: "numeric",
        hour: "2-digit",
        minute: "2-digit",
        timeZoneName: "short",
    };
	// update the <text/> elements with the relevant information
    g
        .select(".date")
        .attr("x", xValue)
        .attr("y", y(datum.load))
        .text("" + new Date(datum.timestamp).toLocaleDateString("en-US", options));
    g
        .select(".load")
        .attr("x", xValue)
        .attr("y", y(datum.load) + 12)
        .text("".concat(datum.load.toFixed(2), " ").concat(magnitudeString));
    g
        .select("line")
        .attr("x1", xValue)
        .attr("x2", xValue)
        .attr("stroke-opacity", 1.0);
}

And then to configure this on our container <svg> element:

svg
    .on("mousemove", function (e) { return updateDataViewLine(d3.pointer(e)[0], context); })
    .on("mouseover", function (e) { return updateDataViewLine(d3.pointer(e)[0], context); })
    .on("mouseout", function () {
    svg.select("#data-view-line")
        .attr("opacity", 0.0);
});

(d3.pointer gives us the relevant coordinates for the event within our graph).

Note that we’re not magic here to make this work. We’re getting some x value from the mouse event, translating it into our graph coordinates, and then figuring out which datum from our array of load data this actually corresponds to. That logic is all here:

var datum = data[Math.floor(data.length * (xValue - marginLeft) / (width - marginLeft - marginRight))];

This seems a bit messy at first glance, but it’s also incredibly straightforward, and that trade-off is in this case I think very much worth it. So with all that said and done, we finally have our interactive graph of energy data.

I might be using the term interactive pretty generously. ↩

Using S3 for static site hosting

2023-10-29T12:41:50+00:00

I decided to move my blog from sourcehut pages to AWS because I was interested in using Google Analytics (for now) to track page impressions¹ and also because I was interested in exploring terraform and the current state of declarative infrastructure. This ended up being considerably more complicated than I expected, but I think my expectations were just incorrect, and the complications are probably warranted. Either way, it can be overwhelming to get such a simple thing set up, and I’d like to document the steps and how they all fit together. So far, everything was configured manually through the AWS console, but I’m going to try to move this to terraform shortly.

The three requirements I had for the new setup were

use/enforce HTTPS
provide access to the blog through my top-level fulwiler.me domain
automatically update the site on push to master branch

To meet these requirements I needed a handful of AWS services.

S3 to store the actual site (.html files, etc.)
Route53 to manage DNS for my domain (fulwiler.me)
AWS Certificate Manager to provision certificates to enable HTTPS
Cloudfront to put HTTPS in front of the S3 bucket, which do not by themselves provide HTTPS access to static websites
Identity and Access Management to provision access keys that allow me to publish into S3

Which loosely are related as follows:

link to mermaid chart

After I had created an s3 bucket and configured it to host a static website, I was able to view the site on fulwiler.me. So far so good. Then I tried to reconfigure my DNS so that fulwiler.me would point there. This turned out to be considerably more painful.

The issue is that in order to do so, I needed to convince DNS lookups for fulwiler.me to resolve to my new s3 website endpoint. In DNS, this is known as a CNAME, which basically just points one domain to another. Unfortunately, it’s impossible to CNAME the top-level domain. If my s3 bucket had a static IP address, I would have been able to add an A record to my top-level domain, but s3 doesn’t provide a static IP address. After going down a few other rabbit holes, I realized that the most straightforward approach would be to simply move my domain over to Route53, and use their custom ALIAS records which allow you to point to arbitrary AWS resources. So I transferred my domain out of Google, which in the background is basically just informing ICANN of the canonical owner of the domain, and reconfigured the nameservers to point to a new hosted zone I created in Route53. A hosted zone is a slightly confusing term that seems to just mean DNS configuration, as this is the where you add all your A, MX, CNAME records, etc.

At this point AWS managed fulwiler.me, providing DNS resolution for the domain, and allowed me to point the top level domain to my s3 static website, backed by my new bucket. Other than HTTPS access, I was mostly done.

In order to set up HTTPS, I needed to generate a certificate with ACM, and then add corresponding records to my DNS configuration (through Route53) which prove my ownership of the domain. I couldn’t directly provide my s3 bucket over HTTPS, however, so I needed to set up a cloudfront distribution that simply points to my s3 bucket. After attaching my certificate to my cloudfront distribution², and then updating my ALIAS record in my DNS configuration to point at my new cloudfront resource (instead of my s3 bucket directly), everything worked!

The last step was simply generating AWS access keys with IAM, storing these in a build secret on builds.sr.ht, adding a new (very simple) app to my flake.nix

apps.aws = {
  type = "app";
  program = "${pkgs.awscli2}/bin/aws";
};

and then building + publishing in my build configuration

tasks:
  - build-site: |
      cd fulwiler.me
      JEKYLL_ENV=production nix run .#jekyll -- build

  - publish-site: |
      cd fulwiler.me

      # only run on master
      git diff --exit-code origin/master || complete-build

      nix run .#aws -- s3 sync build/ s3://fulwiler.me

Figuring out how all of these various s3 resources fit together took a bit of time, and the decoupling of everything at this point feels unnecessarily complex for my (very simple) use case, but I’m sure I’ll appreciate the flexibility as I start to get more familiar with the various services.

Sourcehut takes a very principled approach to user tracking, which I appreciate, but seems to also be against any sort of site metrics which feel like two different things. I'd love to move away from Google Analytics to something less invasive, but haven't had the time to look into what the best alternative is.

↩
I ran into issues here because it wasn't clear to me that my cloudfront setup needed to establish fulwiler.me as an alternate domain in order to successfully use my HTTPS certificate.

↩

Tracking my internet speed with nix and systemd: Part II

2023-10-27T15:32:20+00:00

See Part 1 for how to set up systemd with nix in order to periodically run this job.

In order to actually persist the download speeds and view them over time (and potentially alert on them if we’re so inclined), we need to set up prometheus and grafana. This blog post gets us most of the way there, and is again a pretty incredible example of the power of nix/NixOS. I haven’t gotten far enough to use nixops, so I’ll just be spinning up the requisite services (prometheus, grafana, pushgateway) locally.

Prometheus stores the metrics, Grafana gives us nice visualizations of the same, and the pushgateway allows us to submit metrics from our batch speedtest monitoring job (typically, prometheus counters are exposed on long-running servers, and prometheus pulls from these in order to collect the metrics. In our case, we’re running one-off ephemeral jobs, so we need somewhere to store the resulting metrics when we’re done).

Nix allows us to configure prometheus, grafana, and pushgateway directly in our nix module, without needing to resort to .yaml files. I’m not sure I totally agree with this approach, as the .yaml configuration for these services should be source of truth, and duplicating that can lead to inconsistencies and confusing or silent failures. However, the convenience factor when it does just work is pretty great.

{ config, ... }:
{
  options = {};
  config = {
    services.prometheus = {
      port = 3020;
      enable = true;

      pushgateway = {
        enable = true;
        web.listen-address = "localhost:3022";
      };

      scrapeConfigs = [{
        job_name = "pushgateway";
        static_configs = [{
          targets = [
            "127.0.0.1:3022"
          ];
        }];
      }];
    };

    services.grafana = {
      port = 3010;

      protocol = "http";
      addr = "127.0.0.1";
      analytics.reporting.enable = false;
      enable = true;

      provision = {
        enable = true;
        datasources.settings.datasources = [
          {
            name = "Prometheus";
            type = "prometheus";
            access = "proxy";
            url = "http://127.0.0.1:${toString config.services.prometheus.port}";
          }
        ];
      };
    };
  };
}

After a nixos-rebuild, we now have these services up and running, and just need to set up or python application to start sending the data.

I usually have a pretty hard time remembering the correct invocations in nix to get a python application packaged up, and it doesn’t help that my understanding of python packaging itself is pretty limited. But after several attempts, I was able to just set up a basic setup.cfg and pyproject.toml, and then package it up with:

  speedtest-monitor = pkgs.python311Packages.buildPythonApplication rec {
    pname = "speedtest-monitor";
    version = "0.0.1";
    srcs = ./.;
    pyproject = true;
    nativeBuildInputs = [
      pkgs.python311.pkgs.setuptools
      pkgs.python311.pkgs.wheel
    ];
    propagatedBuildInputs = with pkgs.python311.pkgs; [
      speedtest-cli
      prometheus-client
    ];
  };

After that, we expose this as a package in our flake, and then reference it on our module to hook it into systemd.

{
  description = "A very basic flake";

  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
  flake-utils.lib.eachDefaultSystem (system:
  let 
  pkgs = import nixpkgs { inherit system; };
  in
  {
    packages.speedtest-monitor = ...;
  }) // {
    nixosModules.default = { lib, pkgs, config, ... }:
    with lib;
    let
      cfg = config.services.speedtest;
    in
    {
      options = { ... };

      config = mkIf cfg.enable {
        systemd.timers.speedtest = { ... };

        systemd.services.speedtest =
          {
            serviceConfig = {
              ExecStart = "${self.packages."${pkgs.system}".speedtest-monitor}/bin/speedtest-monitor";
              Type = "oneshot";
              User = "root";
            };
          };
      };
    };
  };
  }

And that’s it! We’ve successfuly used nix to spin up a bunch of services through systemd, install our python interpreter as well as dependent libraries from Pypi, and hook everything together in order to start monitoring our download speeds periodically.

All the code for this project is here.

Tracking my internet speed with nix and systemd: Part I

2023-10-24T15:32:20+00:00

I recently upgraded to gigabit internet through Verizon FiOS, for basically no other reason than that having those sorts of download speeds is quite thrilling. Initially, the speeds were pretty comparable to what was advertised, but I wanted to set up a recurring job to start tracking them over time both to make sure it stayed consistent but also out of genuine curiosity about the variability of the network. I’d also been meaning to dig into nixos modules for quite some time, and this seemed like the perfect opportunity to do so.

The workflow for modules is pretty simple. Basically, modules expose a set of knobs that can be set in your nixos configuration, that then push functionality into the resulting materialized system. The official docs in this case are pretty illuminating, but I’ve trimmed their example a bit just to highlight the basic enable + mkIf enable ... workflow typical to modules.

{ lib, pkgs, config, ... }:
{
  options = {
    services.hello = {
      enable = lib.mkEnableOption "hello service";
    };
  };

  config = lib.mkIf config.services.hello.enable {
    systemd.services.hello = {
      wantedBy = [ "multi-user.target" ];
      serviceConfig.ExecStart = "${pkgs.hello}/bin/hello -g'Hello, World!'";
    };
  };
}

If you’re not used to nix, the fact that this works at all is I think pretty amazing, and maybe just slightly unhinged. Basically the module provides two attributes, options and config. options tells the module system which settings to expose in your root configuration, and then config actually pushes stuff into that configuration, usually by relying on an enable flag in options, but this is just convention.

After saving this in hello-world.nix, you can pull it in like so:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs";

  outputs = { self, nixpkgs, ... }: 
    {
      nixosConfigurations."desktop" = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [ 
          ./hardware-configuration.nix
          ./hello-world.nix
          {
            services.hello.enable = true;
          }		
        ]
      };
    };
}

and now after a nixos rebuild you’ll have a fully configured systemd service, which you can inspect with systemctl status hello.service and journalctl -u hello.service.

As with most things nix, I will admit to not knowing several critical details of how this all fits together. For example, the { lib, pkgs, config, ... } part of the module is a bit magical to me, and I’m not even entirely sure what those pkgs really are, or what config is referring to. I’ve taken to copy/paste-ing a bunch with nix, and true nirvana still evades me, but I have found it to be incredibly powerful despite these nagging uncertainties.

So using this basic approach, I wanted to set up a systemd service that periodically invokes speedtest, and sends the results to my prometheus instance. Luckily, speedtest-cli already exists, and invoking it is straightforward (though you do need to add --secure or it won’t work). Luckier still, speedtest-cli already exists in nixpkgs.

To pull in the package, you use a bit of nix magic via python3.withPackages which gives you a python interpreter that already has the packages installed in its site-packages/ directory.

{ lib, pkgs, config, ... }:
let
  python-env = pkgs.python3.withPackages (ps: with ps; [ speedtest-cli ]);
in 
{
  options.services.speedtest = {
    enable = lib.mkEnableOption "speedtest service";
  };

  config = lib.mkIf config.services.speedtest.enable {
    systemd.services.speedtest = {
      wantedBy = [ "multi-user.target" ];
      serviceConfig.ExecStart = "${python-env}/bin/speedtest-cli --secure";
    };
  };
}

After referencing the configured python interpreter in our interpolated ExecStart, nix pulls in the python package and enables us to reference the executable that ends up in the bin/ directory of that store path.

There’s a lot going on here, but ultimately what’s happening feels like how things should work, in a way that makes me question why everything else is so complicated. python3.withPackages involves a lot of nix that I don’t understand, but conceptually I understand the resulting DAG, and the fact that that entire DAG is pulled in through this string interpolated systemd command is pretty remarkable.

See Part II for setting up prometheus and pushing the results.