See Part 1 for how to set up systemd with nix in order to periodically run this job.

In order to actually persist the download speeds and view them over time (and potentially alert on them if we’re so inclined), we need to set up prometheus and grafana. This blog post gets us most of the way there, and is again a pretty incredible example of the power of nix/NixOS. I haven’t gotten far enough to use nixops, so I’ll just be spinning up the requisite services (prometheus, grafana, pushgateway) locally.

Prometheus stores the metrics, Grafana gives us nice visualizations of the same, and the pushgateway allows us to submit metrics from our batch speedtest monitoring job (typically, prometheus counters are exposed on long-running servers, and prometheus pulls from these in order to collect the metrics. In our case, we’re running one-off ephemeral jobs, so we need somewhere to store the resulting metrics when we’re done).

Nix allows us to configure prometheus, grafana, and pushgateway directly in our nix module, without needing to resort to .yaml files. I’m not sure I totally agree with this approach, as the .yaml configuration for these services should be source of truth, and duplicating that can lead to inconsistencies and confusing or silent failures. However, the convenience factor when it does just work is pretty great.

{ config, ... }:
{
  options = {};
  config = {
    services.prometheus = {
      port = 3020;
      enable = true;

      pushgateway = {
        enable = true;
        web.listen-address = "localhost:3022";
      };

      scrapeConfigs = [{
        job_name = "pushgateway";
        static_configs = [{
          targets = [
            "127.0.0.1:3022"
          ];
        }];
      }];
    };

    services.grafana = {
      port = 3010;

      protocol = "http";
      addr = "127.0.0.1";
      analytics.reporting.enable = false;
      enable = true;

      provision = {
        enable = true;
        datasources.settings.datasources = [
          {
            name = "Prometheus";
            type = "prometheus";
            access = "proxy";
            url = "http://127.0.0.1:${toString config.services.prometheus.port}";
          }
        ];
      };
    };
  };
}

After a nixos-rebuild, we now have these services up and running, and just need to set up or python application to start sending the data.

I usually have a pretty hard time remembering the correct invocations in nix to get a python application packaged up, and it doesn’t help that my understanding of python packaging itself is pretty limited. But after several attempts, I was able to just set up a basic setup.cfg and pyproject.toml, and then package it up with:

  speedtest-monitor = pkgs.python311Packages.buildPythonApplication rec {
    pname = "speedtest-monitor";
    version = "0.0.1";
    srcs = ./.;
    pyproject = true;
    nativeBuildInputs = [
      pkgs.python311.pkgs.setuptools
      pkgs.python311.pkgs.wheel
    ];
    propagatedBuildInputs = with pkgs.python311.pkgs; [
      speedtest-cli
      prometheus-client
    ];
  };

After that, we expose this as a package in our flake, and then reference it on our module to hook it into systemd.

{
  description = "A very basic flake";

  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
  flake-utils.lib.eachDefaultSystem (system:
  let 
  pkgs = import nixpkgs { inherit system; };
  in
  {
    packages.speedtest-monitor = ...;
  }) // {
    nixosModules.default = { lib, pkgs, config, ... }:
    with lib;
    let
      cfg = config.services.speedtest;
    in
    {
      options = { ... };

      config = mkIf cfg.enable {
        systemd.timers.speedtest = { ... };

        systemd.services.speedtest =
          {
            serviceConfig = {
              ExecStart = "${self.packages."${pkgs.system}".speedtest-monitor}/bin/speedtest-monitor";
              Type = "oneshot";
              User = "root";
            };
          };
      };
    };
  };
  }

And that’s it! We’ve successfuly used nix to spin up a bunch of services through systemd, install our python interpreter as well as dependent libraries from Pypi, and hook everything together in order to start monitoring our download speeds periodically.

All the code for this project is here.