"touch: not found" in installworld

I've been updating one of my servers from FreeBSD 6.0 to 6.1, and had done a make buildworld a weeek or so ago, but didn't get around to actually installing it at the time. Since then, I cvsup'ed the source again, and the only real change was in /etc/rc.d/jail. I figured I didn't need to buildworld again since it's just a shell script, doesn't get compiled, and is installed by mergemaster (instead of make installworld).

When I did make installworld on my slightly outdated world, it errored out almost right away with: "touch: not found". The FreeBSD FAQ mentions that:

This error does not mean that the touch(1) utility is missing. The error is instead propably due to the dates of the files being set sometime in the future. If your CMOS-clock is set to local time you need to run the command adjkerntz -i to adjust the kernel clock when booting into single user mode.

I wasn't in single user mode, and the clock was correct - however my /usr/src/sys/conf/newvers.sh file was dated later than the world I had built, and that seemed to be causing the error. Using touch(1) to set the date back on that one file to match the world seems to have fixed the problem.

OpenLDAP entryUUID generation

At work I've been fighting with OpenLDAP, hoping to update some really old servers (as in 2.0.xx), and possibly using syncrepl instead of slurpd to replicate between servers. One thing that's been holding me up is that it wasn't obvious at all how entryUUID and entryCSN attributes get generated.

Turns out that if you have lastmod off in your slapd.conf, as I did, that seems to suppress generating those attributes, in addition to the ones that are actually documented (modifiersName, modifyTimestamp, ...). I had left lastmod off because I was getting info from an Netscape Directory Server, which already put in that info, and didn't want OpenLDAP messing with it. Turns out that wasn't a concern, and from what I can tell from the sourcecode, OpenLDAP only adds modification attributes if they don't already exist.

Overall I've been pretty frustrated with OpenLDAP documentation through the years, and reading the sourcecode isn't much better, since comments at least in the files I looked at are sparse to nonexistant.

Portaudit and ezjail

Portaudit is a handy utility for FreeBSD that lets you know if any of your installed ports has a known security vulnerability. Part of the install puts a script in /usr/local/etc/periodic/security, which adds a report on ports that should be updated, to the daily security e-mail the system sends to root.

If you have jails setup on your machine, they may have their own ports installed which you'd probably also want checked by portaudit. The brute-force way to do it would be to install separate copies of portaudit inside each jail, and keep an eye on separate daily security e-mails from each jail looking for problems.

In my case, I've been running jails setup by ezjail, and didn't want to install portaudit over and over again. Instead, I came up with this minor shell script that checks each ezjail. If you save it as /usr/local/etc/periodic/security/410.portaudit_ezjail, then it'll run each day, right after the main portaudit periodic script that updates the vulnerability db and checks the main machine, and include the output in the main machine's security e-mail.

#!/bin/sh

#
# Run portaudit against packages installed in ezjails, as
# a periodic security job.
#
#
# 2006-05-05 Barry Pederson <bp@barryp.org>
#


JAIL_CONFIGDIR="/usr/local/etc/ezjail"
PACKAGE_DIR="/var/db/pkg"

# If there is a global system configuration file, suck it in.
#
if [ -r /etc/defaults/periodic.conf ]; then
    . /etc/defaults/periodic.conf
    source_periodic_confs
fi

case "${daily_status_security_portaudit_enable:-YES}" in
    [Nn][Oo])
        ;;
    *)
                for jailname in `ls $JAIL_CONFIGDIR`
                do
                    . "${JAIL_CONFIGDIR}/${jailname}"
                    eval rootdir=\"\$jail_${jailname}_rootdir\"    

                    echo
                    echo "Jail: $jailname"
                    echo "-------------------------"

                    echo "ls ${rootdir}${PACKAGE_DIR} | xargs portaudit" |
            su -fm "${daily_status_security_portaudit_user:-nobody}"
                done
        ;;
esac

I have to admit I'm not too fluent with shell scripting, and would have been much more comfortable writing it in Python, but that's probably a bit of overkill in this case.


Doh! As soon as I finished writing this, I happened to check the ezjail website, and found a link to jailaudit, by Philipp Wuensche which looks to do a similar thing but with more options, and has been submitted as a port.

Sharing a ports tree with ezjail

ezjail's ezjail-admin utility has a -P option to the update subcommand that causes it to fetch/update a ports tree into the basejail directory that all jails then share. However, if your machine already has a /usr/ports tree, that seems like a big waste of space. Why not have jails use that existing tree through mount_nullfs the same way the basejail is shared?

One of the files ezjail creates along with a new jail is /etc/fstab.jailname, that contains something like:

/data/jails/basejail /data/jails/jailname/basejail nullfs ro 0 0

(/data/jails was where I setup ezjail to store my jails)
Just add another line to that file like:

/usr/ports /data/jails/jailname/usr/ports nullfs ro 0 0

And make sure your jail has an empty /usr/ports directory (which is something you can put in a flavour if you're going to be doing this often). When your jail starts, you'll have a readonly view of the main machine's ports tree.

To keep both jailed and non-jailed systems from trying to put any port-building working-directories or downloaded distribution files in /usr/ports, the /etc/make.conf files (both the "real" one and the ones inside jails) should contain something like:

WRKDIRPREFIX=           /var/ports
DISTDIR=                /var/ports/distfiles
PACKAGES=               /var/ports/packages

ezjail's default flavour takes care of the jailed copies of this for you. If you make your own flavour, be sure it includes a similar /etc/make.conf

One last trick... If you're using portupgrade, run portsdb -u after updating your ports from your non-jailed environment. That way, if you're also running portupgrade inside the jail, it won't see its INDEX db as being out of date and complain that it can't fix it because the filesystem is readonly. On my machines I update using portsnap (a great tool BTW, also available to older BSDs as a port) with this trivial script:

#!/bin/sh

portsnap fetch
portsnap update

#
# Also update portupgrade database
#
portsdb -u

NAT and Jails

In experimenting with jails, I've had a need to put them on machines in which I didn't have extra public IP addresses to assign to the NIC. Turns out you can easily assign private addresses to an interface, and setup NAT (Network Address Translation) to allow the jails access to the rest of the world.

The loopback interface lo0 seems to work pretty well for this. On one machine I put ezjail on, I just picked the IP block 10.51.50.x out of my hat, and added an alias address on-the-fly with this command:

ifconfig lo0 alias 10.51.50.1 netmask 255.255.255.255

To make it happen at boot time, add this to /etc/rc.conf:

ifconfig_lo0_alias0="inet 10.51.50.1 netmast 255.255.255.255"

To setup FreeBSD's PF to NAT to the 10.51.50.x block, this went into /etc/pf.conf, after any scrub directives but before any block/pass type rules:

nat on $ext_if from 10.51.50.0/24 to any -> $ext_if

Reload the PF configuration with:

pfctl -f /etc/pf.conf

On another machine, I did mostly the same setup, except for using 127.x.x.x numbers. Not sure if there's any advantage one way or the other, both machines seemed to work pretty much the same.

ezjail really does make jails easy

Virtualization is something I've been interested in for some time, dabbling with VMWare on Windows, and eagerly awaiting Xen+BSD and AMD's Pacifica-enabled chips. FreeBSD's jail feature gives many of the same benefits but with relatively little overhead, as long as you're interested in working with the same version of FreeBSD in your "virtual" system as your "host" is running. Jails are a great way to isolate software - for security reasons, to run different versions of the same package, or just to allow yourself a sandbox to mess with that you can easily wipe out and recreate in a few seconds.

The man page for jail describes how to setup a jail by hand, which seems a bit involved. Luckily I stumbled across ezjail, which makes creating jails a breeze. Once it's setup, you can create and "boot" a fully functioning jail with just three commands. ezjail arranges things so most of the FreeBSD userland is shared between the jails, and the files unique to each jail take up as little as 2mb.

The initial setup is basically:

  1. install the port in sysutils/ezjail
  2. Add ezjail_enable="YES" to /etc/rc.conf
  3. edit /usr/local/etc/ezjail.conf to set where you want your jails created. (In my case I used /data/jails)
  4. make sure your /usr/src tree is complete
  5. run ezjail-admin update

That last command can take a lot of time (maybe hours), since it does a full make buildworld, make installworld. If you've already built your world, there's a -i parameter for skipping that step and just doing the make installworld.

Once that's all done, in your jail directory there is a basejail which contains about 130+mb of files that will be shared between jails, newjail which is a skeleton containing about 2mb of files that gets copied to any new jails you create, and flavours which is basically another set of skeleton directories that get copied over the newjail skeleton when your jail is created.

At this point, you can create and boot a jail with:

  1. ifconfig lo0 alias 127.66.0.1 netmask 255.255.255.255 or similar to give one of your network interfaces an IP the jail can use.
  2. ezjail create myjail 127.66.0.1 creates a new directory (/data/jails/myjail in my case) that's a copy of newjail and sets a few other things up.
  3. /usr/local/etc/rc.d/ezjail.sh start myjail

At this point the jail is up and running. You can "log into" it by first finding out the integer id of the jail with jls, and then running jexec <jail-id> /bin/sh

There are a few things that are missing in this barebones install, mainly no /etc/resolv.conf so domain name lookups don't work, no /etc/localtime so time in the jail shows as UTC. You can fix these problems and add your own customizations easily by using a flavour (don't mess with the newjail template directory).

You can stop and wipe out your jail with

/usr/local/etc/rc.d/ezjail.sh stop myjail
ezjail-admin delete -w myjail

Then, to make a new flavour and make a jail using that flavour, something like

cd /data/jails/flavours
cp -pr default myflavour
cd myflavour/etc
cp -p /etc/resolv.conf .
cp -p /etc/localtime .
ezjail-admin create -f myflavour myjail 127.66.0.1
/usr/local/etc/rc.d/ezjail.sh start myjail

At this point, you've created a new jail with your customizations, and would use jls again to find the jail-id, and jexec to start a shell inside the running jail.

A flavour may also contain packages you wish to install upon jail creation, and commands to execute when the jail is created. Check out the ezjail.flavour file in your flavour directory. I've used it to install common useful things like bash, vim, gmake, and libiconv and gettext which take a long time to build that you don't want to repeat for every jail.

Viewing a man file

This is one of those little things that I just want to jot down for myself so I have it written down until I learn it for good. To view a man file, that's not installed in the regular man file locations, just run

nroff -man filename | more

Stupidly simple, but unfortunately not mentioned in the manpage for man.

Returned from PyCon

Got back from PyCon 2006, in mostly one piece. Picked up a terrible cold at the conference, I suppose scrounging food off the same buffet tables as 400 other people wasn't the most hygenic thing in the world.

Attended the mainly web-oriented sessions, came away very impressed with Django. I had sort of blown it off before because I didn't like the look of the templating language, and the ORM seemed weird. But after seeing what's coming in the Removing the Magic branch, I think it will be much much nicer. Was even inspired to spend the little time I had there Monday morning and afternoon sprinting with the Django guys, but I don't see how one can sprint effectively in such a short time with the limited knowledge of the codebase I had. Maybe if I go next year ... and I know more Django ... and can spend more than a day there, then I could accomplish something useful during the time.

The TurboGears guys demonstrated some nice things with AJAX widgets, but the SQLObject part of TG has given me trouble in the past when working with an existing DB, and seems to get in the way more than it helps. Even so, the TG guys, and Ian Bicking seemed pretty cool, so I hope they polish things up a bit more. Maybe SQLObject 2 will be the answer, or maybe a switch to SQLAlchemy (which wasn't represented at the conference), would make TG a nicer environment to work in.

I've been struggling with Zope for some years now. From a user standpoint I guess it's OK, from a programmer standpoint it's a nightmare, both 2.x and 3.x. The documentation and community attitude have rubbed me wrong for a long time. I attended a couple Zope sessions at the conference, but didn't hear anything to inspire me to keep up with it. I'll probably switch what little Zope things I have going to Django/TurboGears/CherryPy/whatever.

The PyParsing presentation on writing an adventure game was interesting, wish I could have attended the more in-depth one but it conflicted with a Django session. PyParsing looks to make a hard job pretty easy, and I'd love to play with it somewhere.

The party at NerdBooks had some decent food, they had a pretty deep selection of books, and the prices on some of the things I looked up were much better than Amazon. Will definitely look there next time I need something.

Lastly, I hope Django or someone who was at the sprint uses the codename "Vacuum Assassin" somewhere. That would just be too cool.

Self-Signed SSL Certificates

Quite often I find myself needing to generate self-signed certificates for use with OpenSSL. There are only three steps required...

Generate a key file, named ssl.key for example:

openssl genrsa -out ssl.key 1024

Generate a Certificate Signing Request for the key, named ssl.csr in this example. You'll be asked a bunch of questions, when asked for Common Name (eg, YOUR name) be sure to enter the domain-name you're making the certificate for (such as www.foobar.edu).

openssl req -new -key ssl.key -out ssl.csr

Generate a signed certificate given the request and key, valid for 10 years (3650 days) and named ssl.crt in this example. When you're done, the ssl.key and ssl.crt files are what you usually need to install in your server.

openssl x509 -req -days 3650 -in ssl.csr -signkey ssl.key -out ssl.crt

As a bonus, here's how to view the contents of a certificate file named ssl.crt

openssl x509 -in ssl.crt -text

mod_python segfault fixed

Just as a followup, it seems the segfault in mod_python on FreeBSD I mentioned before was found and fixed. Turns out to not be any kind of pointer/memory corruption like I thought, but rather a mishandled return code from an APR (Apache Portable Runtime) function. Oh well, I got to play with gdb, ddd, and valgrind a bit, which is good stuff to be familiar with.