Trying poo-DRE-eh — uh, poudriere

This is my poudriere tutorial. There are many like it. But this one is mine. I built mine with resources like the BSDNow tutorial and the FreeBSD Forums tutorial. While all poudriere tutorials are inadequate, mine is inadequate in new and exciting ways. I’m writing it for my benefit, but what the heck, might as well post it here. (If you read this and say “I learned nothing new,” well, I warned you.)

Your package building system must run the newest version of FreeBSD you want to support. I have 8, 9, and 10 in production, so my package builder needs to be FreeBSD 10 or newer. I’m using FreeBSD 11, because I’m running my package builder on my desktop. I upgraded this machine to the latest -current and updated all my packages and ports.

Poudriere works on either UFS or ZFS partitions. I have copious disk, so I’ll dedicating two of them to poudriere. (This means I’ll have to blow away my install later when I start experimenting with disks, hence this blog posting.) I use disks ada2 and ada3.

First, eradicate anything already on those disks.

# gpart destroy -F ada3
ada3 destroyed
# gpart destroy -F ada2
ada2 destroyed
#

Now I can create new GPT partitions. Each disk needs one partition for ZFS, covering the whole disk. You’ll find lots of discussion about partitioning disks with 512-byte sectors versus those with 4KB sectors, and the need for carefully aligning your disk partitions with the underlying sector size. The easy way around this is to duck the whole issue and assume 4KB sectors, use a null GEOM layer to align everything to the disk’s expectations, and ZFS the null layer. (If you KNOW the sector size of your disk, you can simplify the below, but remember, disks lie about their sector size just like they lie about geometry. It’s safest to gnop all the things.)

# gpart create -s gpt ada2
# gpart create -s gpt ada3
# gpart add -a 4k -t freebsd-zfs -l disk2 ada2
# gpart add -a 4k -t freebsd-zfs -l disk3 ada3
# gnop create -S 4096 /dev/gpt/disk2
# gnop create -S 4096 /dev/gpt/disk3
# zpool create poudriere mirror /dev/gpt/disk2.nop /dev/gpt/disk3.nop

I now have a filesystem to build packages on. Let’s get me a key to sign them with.

# mkdir -p /usr/local/etc/ssl/keys
# mkdir -p /usr/local/etc/ssl/certs
# cd /usr/local/etc/ssl
# openssl genrsa -out keys/mwlucas.key 4096
# openssl rsa -in keys/mwlucas.key -pubout > certs/mwlucas.cert

Installed the latest poudriere from /usr/ports/ports-mgmt/poudriere-devel. While the configuration file is lengthy, you don’t need to set many options.

ZPOOL=poudriere
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/poudriere
POUDRIERE_DATA=${BASEFS}/data
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/mwlucas.key
URL_BASE=http://storm.michaelwlucas.com/

I set the CHECK_CHANGED options because when I update my ports tree, I want to know about changed options before I build and deploy my packages. I set the URL_BASE so I can view the build logs on my web server.

The build process uses its own make.conf file, /usr/local/etc/poudriere.d/make.conf, where I set some very basic things. The only mandatory setting is WITH_PKGNG. You could also have a separate make.conf for each individual jail, but I want all of my packages consistent, so I only use the central file.

WITH_PKGNG="yes"
WITHOUT_X11="yes"
WITHOUT_HAL="yes"
WITHOUT_NLS="yes"

Get a ports tree just for poudriere. I could use the ports tree on the package builder, but it’s possible that ports on the builder might differ from what I want for the packages. It’s best to keep everything tidy.

# poudriere ports -c
====>> Creating default fs... done
====>> Extracting portstree "default"...
Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
Fetching public key from your-org.portsnap.freebsd.org... done.
Fetching snapshot tag from your-org.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
...

Now create a jail to build packages in.

At first blush, you might give the jail any random name. Using the same jail standard as the FreeBSD package builder eases deployment, however. The official naming standard combines operating system, CPU architecture, operating system version, and word size, like so. For example, 32-bit FreeBSD 10 on Intel-type processors is freebsd:10:x86:32, while the 64-bit version is freebsd:10:x86:64. (See pkg-repository for details.)

FreeBSD’s official package builds occur on the oldest supported version of a release. Packages for 9-stable are built on 9.1, so I do:

# poudriere jail -c -j freebsd:9:x86:32 -v 9.1-RELEASE -a i386

Then create jails for the other two releases I support.

# poudriere jail -c -j freebsd:9:x86:32 -v 9.1-RELEASE -a amd64
# poudriere jail -c -j freebsd:10:x86:64 -v 10.0-RELEASE -a amd64

I only run the amd64 version of FreeBSD 10, because I don’t want to build i386 packages forever.

This takes a little while, so start it and walk away. I copied these lines into a shell script and went to lunch.

Note that these are not actual jails. They will not show up in jls(8). Technically, they’re chroots. (I’m not sure why poudriere calls them jails – maybe they were originally such but the need for a full jail went away, or they’re intended to become full jails later on. I’m sure there’s a perfectly sensible reason, however.) You can list your package-building jails only via poudriere.

# poudriere jail -l
JAILNAME VERSION ARCH METHOD PATH
freebsd:9:x86:64 9.1-RELEASE amd64 ftp /poudriere/jails/freebsd:9:x86:64
freebsd:9:x86:32 9.1-RELEASE i386 ftp /poudriere/jails/freebsd:9:x86:32
freebsd:10:x86:64 10.0-RELEASE amd64 ftp /poudriere/jails/freebsd:10:x86:64

Before building any packages, I want to update all the jails to the latest version. As I’ll need to do this before every package build, I script it. I also add the command to update poudriere’s ports tree.

#!/bin/sh

#update ports tree
poudriere ports -p default -u

#Update known builder jails to latest version
poudriere jail -u -j freebsd:9:x86:32
poudriere jail -u -j freebsd:9:x86:64
poudriere jail -u -j freebsd:10:x86:64

I must run this script every time I update my packages.

Now to determine which packages I want to build. In my case, I want to build only packages that I can’t get from official FreeBSD sources. This means things like freeradius and Apache with LDAP support and PHP with Apache support. Simple enough. I created /usr/local/etc/poudriere.d/pkglist.txt containing:

#web services - need LDAP & Apache PHP module
www/apache22
lang/php5
#network - need LDAP & SMB
net/freeradius2
net/openldap24-server

Now the fun part: setting the build options for these ports.

# poudriere options -cf pkglist.txt

This takes you into a recursive make config for all of your selected packages. If you know exactly which ports you must configure to get a properly built package, you could specify ports by name. Unfortunately, I always forget to add LDAP to some dependency, so I walk through all the configurations adding my various options.

You can now build your packages. I want to update all of my packages simultaneously, so I wrote a trivial shell script to build packages.

#!/bin/sh

#Update known builder jails to latest version

poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:9:x86:32
poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:9:x86:64
poudriere bulk -f /usr/local/etc/poudriere.d/pkglist.txt -j freebsd:10:x86:64

Walk away. Or, if you prefer, set up a web server so you can see the build progress and deliver packages to clients. I used apache22 and created /usr/local/etc/apache22/Includes/poudriere.conf containing:

NameVirtualHost *:80

ServerAdmin mwlucas@minetworkservices.com
DocumentRoot /poudriere/data
ServerName storm.blackhelicopters.org


Options Indexes FollowSymLinks
AllowOverride AuthConfig
Order allow,deny
Allow from all


Yes, I could have hacked up httpd.conf to set DocumentRoot, but I prefer to leave package-created files alone if possible.

Let’s turn to the client while this builds. The client needs a repository configuration file and a copy of the build certificate. Here’s the configuration for my local repository, named mwlucas.

mwlucas: {
url: "http://storm.blackhelicopters.org/packages/${ABI}-default/",
mirror_type: "http",
signature_type: "pubkey",
pubkey: "/usr/local/etc/ssl/certs/mwlucas.cert",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}

By using the standard jail names, I was able to use the ${ABI} variable in my repository path. This saves me from needing to update my repository configuration every time I upgrade.

I copy my certificate to the directory specified in the pubkey option.

You should be able to browse to the URL given as your repository. If the URL doesn’t work, you misconfigured your web server.

Check your repository configuration with pkg -vv. You should see all configured repositories.

# pkg -vv
...
Repositories:
FreeBSD: {
url : "pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest",
enabled : yes,
mirror_type : "SRV",
signature_type : "FINGERPRINTS",
fingerprints : "/usr/share/keys/pkg"
}
mwlucas: {
url : "http://storm.blackhelicopters.org/packages/freebsd:10:x86:64-default/Latest/",
enabled : yes,
mirror_type : "HTTP",
signature_type : "PUBKEY",
fingerprints : "/usr/share/keys/pkg",
pubkey : "/usr/local/etc/ssl/certs/mwlucas.cert"
}

Two repositories? Good. But can you actually access the repository? A pkg update should pull down the digests for your new repo.

# pkg update
Updating repository catalogue
digests.txz 100% 1928 1.9KB/s 1.9KB/s 00:00
packagesite.txz 100% 10KB 10.0KB/s 10.0KB/s 00:00
Incremental update completed, 23 packages processed:
0 packages updated, 0 removed and 23 added.

A quick check will verify that the private repo has 23 packages.

Now for the weakest part of pkgng: actually using your repository for select packages. If I built everything, I’d just disable the FreeBSD repository. But I want to only build packages that differ from the default.

pkg searches for packages in each repository in order. At the moment, repositories appear in alphabetical order. I could rename my repository so that it appears before FreeBSD. pkg would search my repo for packages, and if they didn’t exist there check the FreeBSD repo.

This would be ideal. But alphabetical repository ordering is not guaranteed. The only way to ensure the packages install from the correct repository is to tell each individual FreeBSD server where it should get specific packages. This kind of sucks, but it’s still an improvement on pkg_add. (I hear that repository handling should improve in future versions of pkg.)

Use the -r flag to specify a repository with pkg. For example, let’s search my private repo for an Apache package.

# pkg search -r mwlucas apache
apache22-2.2.26

The package is there. Excellent. Install it.

# pkg install -r mwlucas apache22
Updating repository catalogue
The following 8 packages will be installed:

Installing expat: 2.1.0 [mwlucas]
Installing perl5: 5.16.3_7 [mwlucas]
Installing pcre: 8.34 [mwlucas]
Installing openldap-client: 2.4.38 [mwlucas]
Installing gdbm: 1.11 [mwlucas]
Installing db42: 4.2.52_5 [mwlucas]
Installing apr: 1.4.8.1.5.3 [mwlucas]
Installing apache22: 2.2.26 [mwlucas]

The installation will require 94 MB more space

19 MB to be downloaded

Proceed with installing packages [y/N]: y

Take careful note of this list of dependency packages installed from your repository.

Here you have to decide how you want to upgrade your server. All packages that you want to upgrade from your repo need to be marked as such. How many of these dependency packages must be upgraded via your repo? That’s a good question. If you took note of which packages you had to change the configuration of, way back in the beginning, you could label only those changed packages as requiring your repo. I didn’t do that. If this was a brand-new machine I would install packages from my repo first and mark everything installed as requiring my repo. I didn’t do that. Instead, I’m going to mark all packages required by apache22 as belonging to my repo, much like this.

# pkg annotate -Ay apache22 repository mwlucas
apache22-2.2.26: added annotation tagged: repository
# pkg annotate -Ay pcre repository mwlucas
…

Repeat this for each package installed by this package install.

The repository tag appears when you run pkg info on a specific package.

# pkg info pcre | grep repository
repository : mwlucas

pkg upgrade will now only use your repository for the specified package.

Is this long and complicated? Sure. But it beats the snot out of maintaining three separate package repositories under the old package system.