NixOS and mediagoblin: Part II

Last time on the blog, I started the process of packaging mediagoblin into a Nix package, and this time we are continuing on our journey.

Switching to v0.10.0 for mediagoblin

One thing that I discovered while I was working on this project was that there are release tags for mediagoblin. So, I decided to switch to using this tag instead of the stable branch, which will mean our build is more reproducible.

src = fetchgit {
  url = "https://git.savannah.gnu.org/git/mediagoblin.git";
  rev = "v0.10.0";
  sha256 = "13s66snz9710xr7s2icnmmqb4756fq369gn0qa6cwja4xr25sjd4";
  fetchSubmodules = true;
};

Fixing The First Build Problem

Last time, our build script looked like this:

buildPhase = ''
  cd $src
  ./bootstrap.sh
'';

And I was getting an error about how aclocal wasn’t found.

So, I decided that it was time to clone the repository and have a look at the bootstrap script myself. After I ran the git clone command and changed into that directory, I read the bootstrap.sh file and found this:

#!/bin/sh

set -e

aclocal -I m4 --install
autoreconf -fvi

git submodule init
git submodule update

So, the git submodule commands are taken care of with our packaging, and the other commands are from autoconf and automake, which are taken care of in Nix using something called the autoreconfHook. So, as it turns out, we don’t need the bootstrap script at all!

The ./configure and make command

So, the next command that we were told to run is a ./configure command, that takes a VIRTUALENV environment variable to configure the Python environment. Nix’s mkDerivation assumes that the configure phase of building something is just the ./configure command. Additionally, the make command is assumed for the build phase.

Therefore, we don’t need to invoke either commands in our build, since they are the default. I might need to bring back the VIRTUALENV version of the config command later, but I’ll tackle it then.

So, instead of redefining configure or build phases, all you have to do is add the autoreconfHook to the buildInputs argument of mkDerivation.

{ autoreconfHook, fetchgit, stdenv }:

stdenv.mkDerivation rec {
  name = "mediagoblin-${version}";
  version = "stable";

  src = ...;

  buildInputs = [
    autoreconfHook
  ];
  meta = ...;
}

So, when I build this, I get the following error:

configure: error: --with-python3 given but no acceptable python3 could be found

So, at this point, the autoconf error is for a missing python dependency. I worked on that next.

Adding Python Dependencies

While I was looking at the bootstrap.sh script, I realized that there was Python code in the repository, when I remembered that mediagoblin stated that it’s implemented in Python for their backend. Once I knew that I was going to be working on Python depedencies, I started hunting for a requirements.txt file. However, I didn’t find it; I found a setup.py file instead.

After looking through it, I found a two places that it declares deps: for Python 2 and for Python 3. I took the list and pasted it into my Nix file, changed the syntax to match, and looked up the names of the Python packages as they appear in Nix instead of PyPI, and I ended up with this:

let
  my-python-packages = python-packages: with python-packages; [
    alembic
    Babel
    bcrypt
    celery
    certifi
    configobj
    dateutil
    email_validator
    exifread
    gst-python
    itsdangerous
    jinja2
    jsonschema
    lxml
    markdown
    oauthlib
    PasteDeploy
    pasteScript
    pillow
    PyLD
    pytest
    pytest_xdist
    pytz
    requests
    six
    sphinx
    sqlalchemy
    unidecode
    virtualenv
    waitress
    webtest
    werkzeug
    wtforms
  ];
  python-with-my-packages = python3.withPackages my-python-packages;
in stdenv.mkDerivation rec {

The two variables that are bound in this let statement show a common pattern for adding Python to a Nix package. The python3.withPackages function call takes a function that takes a list of Python packages and selects a subset. That’s what the my-python-packages does: it defines that filter function.

Python version mismatch

When I ran the build this time, I got this error:

error: cheetah-2.4.4 not supported for interpreter python3.8

So, it seems that a python package is missing for the Python 3.8, which is the assumed Python version when you use python3 in Nix currently. Let’s see what happens when we change python3 to python37.

error: cheetah-2.4.4 not supported for interpreter python3.7

Oh boy. And I thought this was going to be easy.

The thing I did next was to see where cheetah-2.4.4 was in the official nixpkgs collection, if anywhere. I ran nix search python cheeta and I got these results:

* nixpkgs.python27Packages.TurboCheetah (python2.7-TurboCheetah)
  TurboGears plugin to support use of Cheetah templates

* nixpkgs.python27Packages.cheetah (python2.7-cheetah)
  A template engine and code generation tool

* nixpkgs.python37Packages.TurboCheetah (python3.7-TurboCheetah)
  TurboGears plugin to support use of Cheetah templates

* nixpkgs.python38Packages.TurboCheetah (python3.8-TurboCheetah)
  TurboGears plugin to support use of Cheetah templates

OK, so cheetah has not made it to the Python 3 side of nixpkgs.

So, this blog post has become long enough, so I’m going to do some investigation to help solve this problem, and report back next week.

Thanks for reading!