- Title:
- keeping my repo tidy with lintpkgsrc
- Authors:
- Paolo Vincenzo Olivo
- Date:
- Topics:
- Pkgsrc
- Id:
- 2k6pea
■ Leftovers and leaves
Since I covered my use case for pkg_comp already, today I wanted to address another question, which is 'how to keep things tidy'.
By default, pkgsrc wil store locally built binaries in the PACKAGES directory, which usually corresponds to $PKGSRCDIR/packages/All. Previously fetched distfiles are kept for later use in inside DISTDIR ($PKGSRCDIR/distfiles) instead, unless `make distclean' is executed in the package directory.
As new releases come and go, both PACKAGES and DISTDIR will begin to be populated with multiple versions of the same files, most of which are left there collecting dust.
All these old/orphaned/unused files tend to occupy a non-negligible amount of space over the time, and make it harder to skim through the repo and the distfiles directories, in search of whatever I may need.
The problem becomes all the more relevant on trunk, where the turnover of new package versions is kept constantly high.
My former solution consisted in a bunch of half-baked scripts which searched for multiple versions of the same files (dropping older ones) and for leftovers of uninstalled software.
And it worked, with the limitations imposed by my limited skills. More recently however, as I began sharing my pkgsrc packages for Slackware, I felt the need of a proper solution, capable of keeping my repo devoid of cluttern.
I discovered and utility, which somehow I had managed to miss for all this time, thanks to a post on an Italian tech blog covering the setup of pkgsrc on Slackware. [1]
I'm referring to pkgtools/lintpkgsrc [2], which turned out to be the no.1 swiss army knive of pkgsrc repo maintenance.
[1] <https://wiki.golem.linux.it/Pkgsrc_su_Slackware> [2] <https://pkgsrc.se/pkgtools/lintpkgsrc>
> DESCRIPTION > lintpkgsrc tries to verify the entire contents of the pkgsrc tree. > lintpkgsrc uses simple regular-expression matching for verifying > files that make up a pkgsrc directory. Note that it does NOT > implement a complete Makefile parser.
I recommend reading the manpage carefully. The package provides quite a handful of cool functions to manage a pkgsrc repo (and also get you out of trouble), but they have to be used wisely.
As a starter `-i' checks the version of each installed package against the current version in pkgsrc.
■ Remove outdated <leaf> packages
According to the manual, lintpkgsrc(1) supports a `-r' switch which most closely resembled what I was looking for.
-r Remove distfiles which are not referenced from any > package's distinfo file, or have an incorrect checksum > (usually due to an interrupted download), and any old > binary packages. To remove any given type, additionally > specify the -o, -m, or -p flags respectively.
Now, my only problem with that is 'and any old binary packages':
Supposing in fact that I finally managed to compile and install Firefox (hereafter referred to as 'Nightly') version 1562, after hours and hours of waiting and a substantial electricity consumption, and then Nightly 1563 comes out, lintpkgsrc deletes my previous package, a bug is found in 1563 and I'm unable to roll-back?
And what if I wanted to keep 1562 and skip few versions in order to save time, while still being able to install/uninstall the package at will?
To address this issue I wrote the small script you can see below. It asks lintpkgsrc to look for outdated binaries in the repo, using with the `-p' option. A file list containing deprecated packages is hereby generated and confronted with the package database: binaries of installed packages are retained even if outdated, while all the other are discarded. In other words, only orphaned 'leaf' binaries are purged.
```
#!/bin/sh
PREFIX=/usr/pkg LINTDIR=${PKGSRCDIR}/lint PKGDBDIR=${PREFIX}/pkgdb PACKAGES=${PKGSRCDIR}/packages
set -e
# directory checks if [ ! -d "${PKGSRCDIR}" ]; then echo "PKGSRCDIR not set" exit 1 else [ -d ${LINTDIR} ] || \ ( echo 'creating lint dir' && mkdir ${LINTDIR} ) fi
# clean up previous logs for f in pkgs old leafs do [ -f "${f}" ] && rm ${LINTDIR}/${f} done
# scan for outdated binaries printf "Looking for obsolete packages...\n" lintpkgsrc -K $PACKAGES -p \ | grep -v 'Cannot extract\|Bogus' \ | sed -n -e 's/^.*All\///p' \ > ${LINTDIR}/pkgs
# prune leaf binaries and orphaned sources printf "Creating file lists...\n" cat ${LINTDIR}/pkgs | while read p; do pbase=`basename $p .tgz` if [ -d ${PKGDBDIR}/${pbase} ]; then echo "$p" >> ${LINTDIR}/old else echo "$p" >> ${LINTDIR}/leafs fi done
printf "Pruning leaf binaries...\n" cat ${LINTDIR}/leafs | while read in; do rm -f ${PACKAGES}/"$in" done
```
■ Prune deprecated distfiles
A little word on outdated distfiles; they can be easily pruned with:
$ export PACKAGES=/path/to/packages $ lintpkgsrc -or
Notice the additional `-o' option before -r, which reports old distfiles only. `-m' can be added as a further sanity check.
> -m For each current distfile, verify its checksum against > the appropriate pack‐ age's distinfo file. Also report if > any package's distinfo file references the same distfile > with a different distinfo checksum.
A comparable functionality (not covering `-m') is provided also by pkgtools/pkg_distinst, which is a perl script.