RetroBSD Net
Title:
Automating pkgsrc builds with pkg_comp
Authors:
Paolo Vincenzo Olivo
Date:
Topics:
Pkgsrc
Id:
6936e3

A sandboxed package builder.

pkg_comp is an utility to build pkgsrc packages in a fully automated and self-contained manner. It can be seen as a wrapper over cvs/git, the pbulk infrastructure provided by pkgsrc, and sandboxctl(8). This turns out particularly useful for whenever official binaries are not available: packages in sync with latest pkgsrc-current changes, packages linked against a -current userland, packages for tier2 archs or archs added recently and only supported in current (pkg_comp used to be my go to on aarch64), packages with licenses preventing them to be freely redistributed as binary. pkg_comp is also useful to set up a small repo to serve on the local network (a trivial task using ftpd(8) or bozohttpd(8)), if bandwidth and download speed are problem. A third interesting use case is compiling packages with custom build options (and prevent them from being refreshed/replaced by a `pkgin fug`), as well as applying local patches.

■ preliminary tasks

1. Get the sysrc and xsrc source sets matching your system version and unpack them to /usr, for packages needing to be linked against them. See <Chapter 32. Obtaining the sources>.

2. Fetch the release binary sets matching your system version. You can use `sysupgrade fetch`, it will store them automatically at /var/cache/sysupgrade.

3. Set up the release dir which sandboxctl shall use to create the sandbox. As of today, sandboxctl doesn't seem to allow specifying the archive extention, hence .tgz is expected. In NetBSD-9, I got around this by using the following `xz2gz` script, which will convert the archives to gzip and move them to /release/binary/sets

#!/bin/sh tgzdir='/release/binary/sets' xzdir='/var/cache/sysupgrade'

if [ ! -d $tgzdir ]; then mkdir -p $tgzdir fi

for set in ${xzdir}/*.xz do xz -d $set gzip ${xzdir}/*.tar done

for set in ${xzdir}/*.tar.gz do mv $set $(basename $set .tar.gz).tgz done

for set in ${xzdir}/*.tgz do mv $set $tgzdir done

configure pkg_comp

Configuration files for pkg_comp and its sandbox are stored by default at /usr/pkg/etc/pkg_comp.

1. sandbox.conf # The `netbsd-release' sandbox type sets up a NetBSD sandbox on a NetBSD # host using release sets. Sandboxes created in this manner are "clean" in # the sense that they reproduce a freshly installed system and are # completely decoupled from the host system SANDBOX_TYPE=netbsd-release # sandbox path SANDBOX_ROOT=/var/tmp/sandbox # Parameters for a netbsd-release sandbox NETBSD_RELEASE_RELEASEDIR=/release NETBSD_RELEASE_SETS="base comp etc games man misc modules rescue tests text xbase xcomp xetc xfont xserver" # User-provided routines required for building packages that use X libraries # or need to build against system and/or X sources pkg_comp_post_mount_hook() { # # Required to be able to build any packages that uses X and its libraries # if [ -d /usr/X11R7 ];; then mkdir -p ${SANDBOX_ROOT}/usr/X11R7 sandbox_bindfs -o ro /usr/X11R7 ${SANDBOX_ROOT}/usr/X11R7 fi # # Optional, but needed for any packages which need access to system or X sources # if [ -d /usr/src ];; then mkdir -p ${SANDBOX_ROOT}/usr/src sandbox_bindfs -o ro /usr/src ${SANDBOX_ROOT}/usr/src fi if [ -d /usr/xsrc ]; then mkdir -p ${SANDBOX_ROOT}/usr/xsrc sandbox_bindfs -o ro /usr/xsrc ${SANDBOX_ROOT}/usr/xsrc fi exit 0 }

2. local.conf, our local pkg_comp configuration # update your pkgsrc tree whenever invoking a pkg_comp build. # not strictly necessary on quartery branches UPDATE_SOURCES=true # Additional configuration files to support this setup. EXTRA_MKCONF="/usr/pkg/etc/pkg_comp/extra.mk.conf" SANDBOX_CONFFILE="/usr/pkg/etc/pkg_comp/sandbox.conf" # Remote VCS configuration. FETCH_VCS=cvs CVS_ROOT=:ext:anoncvs@anoncvs.NetBSD.org:/cvsroot CVS_TAG=pkgsrc #GIT_URL=https://github.com/NetBSD/pkgsrc.git #GIT_BRANCH=trunk # Host file layout # These directories point to trees managed outside of the # sandbox and are exposed within the sandbox via null mounts. PKGSRCDIR="/usr/pkgsrc" DISTDIR="/usr/pkgsrc/distfiles" PACKAGES="${PKGSRCDIR}/packages" PBULK_LOG="${PACKAGES}/log" PBULK_PACKAGES="${PACKAGES}/pbulk" # Target file layout LOCALBASE=/usr/pkg PKG_DBDIR=/usr/pkg/pkgdb SYSCONFDIR=/usr/pkg/etc VARBASE=/var # Enable interactive pkgsrc development via sandbox-shell. PKG_DEVELOPER=yes # List of packages to build during automatic execution. AUTO_PACKAGES="$(grep -v '^#' '/usr/pkg/etc/pkg_comp/pbulk.list')"

3. extra.mk.conf, additional pkgsrc settings to be used with pkg_comp. # extra # sanity checks for shared libs PKG_DEVELOPER=yes # compatibility pkgdbdir PKG_DBDIR=/var/db/pkg # build on tmpfs WRKOBJDIR=/var/tmp/pkg # cflags #CPUFLAGS+=-O2 -pipe -fomit-frame-pointer -mfpmath=sse -msse2 -march=native #CFLAGS+=-Og -g -DDEBUG=1 #CFLAGS+=-Wfatal-errors # ccache #PKGSRC_COMPILER=ccache gcc #CCACHE_DIR=/var/tmp/ccache # make jobs MAKE_JOBS=$(/sbin/sysctl -n hw.ncpuonline) # PKG_ALTERNATIVES PYTHON_VERSION_DEFAULT=39 LUA_VERSION_DEFAULT=54 PHP_VERSION_DEFAULT=80 RUBY_VERSION_DEFAULT=30 # avoid linking against outdated libs PREFER_PKGSRC?=MesaLib Xft2 Xrandr Xrender expat fontconfig freetype2 glu openssl pixman xcursor #PREFER_PKGSRC=yes # user patches LOCALPATCHES=/usr/pkg/etc/pkg_comp/patches # other SKIP_LICENSE_CHECK=yes #NO_CHECKSUM=yes #ALLOW_VULNERABLE_PACKAGES=yes # PKG_OPTIONS PKG_OPTIONS.seamonkey=dbus webrtc PKG_OPTIONS.packagename=option1 option2 -disabledoption3

4. pbulk.list, list of packages to build. # specify a list of packages to build, one per line # with a category prefix, as they are found in the # pkgsrc tree. wm/ctwm editors/texmaker print/zathura converters/pandoc graphics/sxiv ...

■ start a pkg_comp build

$ pkg_comp -c local auto

That's pretty much all you need. The `local` configuration file, specified *without* the .conf extension, is expected to be found in the standard /usr/pkg/etc/pkg_comp directory. The `auto` command will tell pkg_comp to: 1. optionally checkout or update a pkgsrc tree (if `UPDATE_SOURCE=true`); 2. create the sandbox (using sandboxctl); 3. bootstrap pkgsrc and pbulk; 4. use pbulk to build the given packages; 5. destroy the sandbox.

And is equivalent to: $ pkg_comp -c local fetch $ pkg_comp -c local sandbox-create $ pkg_comp -c local bootstrap $ pkg_comp -c local build <pbulk.list> $ pkg_comp -c local sandbox-destroy

The stages can be invoked separately (e.g. you may just create/destroy the sandbox and build the packages in the middle). The `sandbox-shell` command will chroot you into the sandbox, this is useful if anything goes wrong with the build. The `-o UPDATE_SOURCE=false` can be passed interactively (as any other setting) to skip updating the pkgsrc tree. If you'd like to restrict the set of packages to build during a manually-triggered build, provide those as additional arguments to `auto` (i.e. 'auto pkgin'). This will override the contents of `AUTOPACKAGES` (which was in turn derived from your pbulk.list file). Nevertheless, you may you may also tell pkgcomp to just build individual packages or a series of packages:

$ pkg_comp -c local build package1 package2 package3 ...

■ installing packages

Any successfully built package will be stored in the directory set in `PACKAGES`, usually /usr/pkgsrc/packages/All. The pkg_summary files are also going to be updated, hence the directory can be used without further changes by pkgin to install or upgrade the packages on your host.

Now, all you need to do to use them via pkg_add(8) is:

$ PKG_PATH=file:///usr/pkgsrc/packages/All; export PKG_PATH $ pkg_add pkgname

Or add the repository to repositories.conf in order to use it with pkgin.