The Ports Tree Build Tool

In the previous post I described the small configuration language that will be used the define the ULD ports tree. In this post I outline the tool that I'm building that will take those definitions and hopefully turn them into built software.

The build tool is responsible for taking a description of how a particular piece of software is built, and what it depends on and building it. I have a few goals in mind for the build tool:

Isolated Builds

A common issue with creating new packages is correctly determining the dependencies of the software being packaged. It's not uncommon to see comments on AUR packages pointing out dependencies that were missed because they were picked up from the host system. xbsp-src goes some way towards addressing this by performing builds in a chroot isolated from the host. It calls this the masterdir. However it appears that it still installs all built packages into the masterdir so it's still possible to have undeclared dependencies.

My idea is to construct the chroot from the base system plus the declared dependencies. Each package being build is staged into an isolated file tree that mirrors the structure of the system it will be installed in. For example the build version of the pastel tool looks like this:

pastel/pkg
└── pastel
   └── usr
      ├── bin
      │  └── pastel
      └── share
         ├── bash-completion
         │  └── completions
         │     └── pastel
         ├── fish
         │  └── vendor_completions.d
         │     └── pastel.fish
         ├── licenses
         │  └── pastel
         │     ├── LICENSE-APACHE
         │     └── LICENSE-MIT
         └── zsh
            └── site-functions
               └── _pastel

Given all built package will have a (versioned) directory like that, those file trees can be overlaid with a base tree to construct the chroot. Additionally it can hopefully all be read-only. The overlay file system in the Linux kernel will be used to create this union view.

When an individual package is built and staged into its tree xbps-create will take care of creating the package from it. xbps is the package manager I've chosen for ULD, so it will take care of managing installation, updating, and deletion of packages on a ULD system.

Don't Build as Root

It's preferable to build software as an unprivileged user. Sometimes there is a need to be root at some stage though, such as to set file owners and permissions. One option is something like fakeroot, which uses so LD_PRELOAD hackery to to patch libc functions:

fakeroot works by replacing the file manipulation library functions (chmod(2), stat(2) etc.) by ones that simulate the effect the real library functions would have had, had the user really been root. These wrapper functions are in a shared library /usr/lib//libfakeroot-.so or similar location on your platform. The shared object is loaded through the LD_PRELOAD mechanism of the dynamic loader. (See ld.so(8))

I'd prefer to avoid such hacks though, especially since it can get in the way of other sensible things. The option I'm exploring is inspired by the OpenBSD project where often their tools will be started as root then drop privileges (and often additionally limit functionality with unveil and pledge). The build tool will be started as root but it will spawn builders that run with regular/reduced privileges. The master process can still potentially do things that require root though, if needed.

Fast and Reliable

A system tool requires a systems language, so along with the rest of the ULD tools that are written from scratch the build tool is written in Rust. This means it is able to readily interact at a system level (constructing and mounting the union filesystem, chroot, multi-process builder, etc.) and be fast and correct about it.

The builder will aim to make the most of the resources available to it. That means using all available cores where possible, downloading distfiles in parallel, and removing the need to install any extra runtime (such a scripting language) to use it.

What's Been Done

So far the build tool can read the full ports tree and then come up with a build plan for how to build either the full set of packages, or a named subset. The ports tree is represented as a graph using the petgraph crate. This crate and data structure make traversing the hierarchy to construct a build plan simple and well defined. Using this library means all dependency queries can be done rapidly, in memory, without the need to repeatedly go to the file system or write out state into the ports tree.

I'm currently working on the worker pool that will be sent build jobs to complete. Once that's in place I'll need to start implementing the various build-styles (cargo, GNU configure, etc.).