Portaudit and ezjail

Portaudit is a handy utility for FreeBSD that lets you know if any of your installed ports has a known security vulnerability. Part of the install puts a script in /usr/local/etc/periodic/security, which adds a report on ports that should be updated, to the daily security e-mail the system sends to root.

If you have jails setup on your machine, they may have their own ports installed which you'd probably also want checked by portaudit. The brute-force way to do it would be to install separate copies of portaudit inside each jail, and keep an eye on separate daily security e-mails from each jail looking for problems.

In my case, I've been running jails setup by ezjail, and didn't want to install portaudit over and over again. Instead, I came up with this minor shell script that checks each ezjail. If you save it as /usr/local/etc/periodic/security/410.portaudit_ezjail, then it'll run each day, right after the main portaudit periodic script that updates the vulnerability db and checks the main machine, and include the output in the main machine's security e-mail.


# Run portaudit against packages installed in ezjails, as
# a periodic security job.
# 2006-05-05 Barry Pederson <bp@barryp.org>


# If there is a global system configuration file, suck it in.
if [ -r /etc/defaults/periodic.conf ]; then
    . /etc/defaults/periodic.conf

case "${daily_status_security_portaudit_enable:-YES}" in
                for jailname in `ls $JAIL_CONFIGDIR`
                    . "${JAIL_CONFIGDIR}/${jailname}"
                    eval rootdir=\"\$jail_${jailname}_rootdir\"    

                    echo "Jail: $jailname"
                    echo "-------------------------"

                    echo "ls ${rootdir}${PACKAGE_DIR} | xargs portaudit" |
            su -fm "${daily_status_security_portaudit_user:-nobody}"

I have to admit I'm not too fluent with shell scripting, and would have been much more comfortable writing it in Python, but that's probably a bit of overkill in this case.

Doh! As soon as I finished writing this, I happened to check the ezjail website, and found a link to jailaudit, by Philipp Wuensche which looks to do a similar thing but with more options, and has been submitted as a port.

Sharing a ports tree with ezjail

ezjail's ezjail-admin utility has a -P option to the update subcommand that causes it to fetch/update a ports tree into the basejail directory that all jails then share. However, if your machine already has a /usr/ports tree, that seems like a big waste of space. Why not have jails use that existing tree through mount_nullfs the same way the basejail is shared?

One of the files ezjail creates along with a new jail is /etc/fstab.jailname, that contains something like:

/data/jails/basejail /data/jails/jailname/basejail nullfs ro 0 0

(/data/jails was where I setup ezjail to store my jails)
Just add another line to that file like:

/usr/ports /data/jails/jailname/usr/ports nullfs ro 0 0

And make sure your jail has an empty /usr/ports directory (which is something you can put in a flavour if you're going to be doing this often). When your jail starts, you'll have a readonly view of the main machine's ports tree.

To keep both jailed and non-jailed systems from trying to put any port-building working-directories or downloaded distribution files in /usr/ports, the /etc/make.conf files (both the "real" one and the ones inside jails) should contain something like:

WRKDIRPREFIX=           /var/ports
DISTDIR=                /var/ports/distfiles
PACKAGES=               /var/ports/packages

ezjail's default flavour takes care of the jailed copies of this for you. If you make your own flavour, be sure it includes a similar /etc/make.conf

One last trick... If you're using portupgrade, run portsdb -u after updating your ports from your non-jailed environment. That way, if you're also running portupgrade inside the jail, it won't see its INDEX db as being out of date and complain that it can't fix it because the filesystem is readonly. On my machines I update using portsnap (a great tool BTW, also available to older BSDs as a port) with this trivial script:


portsnap fetch
portsnap update

# Also update portupgrade database
portsdb -u

NAT and Jails

In experimenting with jails, I've had a need to put them on machines in which I didn't have extra public IP addresses to assign to the NIC. Turns out you can easily assign private addresses to an interface, and setup NAT (Network Address Translation) to allow the jails access to the rest of the world.

The loopback interface lo0 seems to work pretty well for this. On one machine I put ezjail on, I just picked the IP block 10.51.50.x out of my hat, and added an alias address on-the-fly with this command:

ifconfig lo0 alias netmask

To make it happen at boot time, add this to /etc/rc.conf:

ifconfig_lo0_alias0="inet netmast"

To setup FreeBSD's PF to NAT to the 10.51.50.x block, this went into /etc/pf.conf, after any scrub directives but before any block/pass type rules:

nat on $ext_if from to any -> $ext_if

Reload the PF configuration with:

pfctl -f /etc/pf.conf

On another machine, I did mostly the same setup, except for using 127.x.x.x numbers. Not sure if there's any advantage one way or the other, both machines seemed to work pretty much the same.

ezjail really does make jails easy

Virtualization is something I've been interested in for some time, dabbling with VMWare on Windows, and eagerly awaiting Xen+BSD and AMD's Pacifica-enabled chips. FreeBSD's jail feature gives many of the same benefits but with relatively little overhead, as long as you're interested in working with the same version of FreeBSD in your "virtual" system as your "host" is running. Jails are a great way to isolate software - for security reasons, to run different versions of the same package, or just to allow yourself a sandbox to mess with that you can easily wipe out and recreate in a few seconds.

The man page for jail describes how to setup a jail by hand, which seems a bit involved. Luckily I stumbled across ezjail, which makes creating jails a breeze. Once it's setup, you can create and "boot" a fully functioning jail with just three commands. ezjail arranges things so most of the FreeBSD userland is shared between the jails, and the files unique to each jail take up as little as 2mb.

The initial setup is basically:

  1. install the port in sysutils/ezjail
  2. Add ezjail_enable="YES" to /etc/rc.conf
  3. edit /usr/local/etc/ezjail.conf to set where you want your jails created. (In my case I used /data/jails)
  4. make sure your /usr/src tree is complete
  5. run ezjail-admin update

That last command can take a lot of time (maybe hours), since it does a full make buildworld, make installworld. If you've already built your world, there's a -i parameter for skipping that step and just doing the make installworld.

Once that's all done, in your jail directory there is a basejail which contains about 130+mb of files that will be shared between jails, newjail which is a skeleton containing about 2mb of files that gets copied to any new jails you create, and flavours which is basically another set of skeleton directories that get copied over the newjail skeleton when your jail is created.

At this point, you can create and boot a jail with:

  1. ifconfig lo0 alias netmask or similar to give one of your network interfaces an IP the jail can use.
  2. ezjail create myjail creates a new directory (/data/jails/myjail in my case) that's a copy of newjail and sets a few other things up.
  3. /usr/local/etc/rc.d/ezjail.sh start myjail

At this point the jail is up and running. You can "log into" it by first finding out the integer id of the jail with jls, and then running jexec <jail-id> /bin/sh

There are a few things that are missing in this barebones install, mainly no /etc/resolv.conf so domain name lookups don't work, no /etc/localtime so time in the jail shows as UTC. You can fix these problems and add your own customizations easily by using a flavour (don't mess with the newjail template directory).

You can stop and wipe out your jail with

/usr/local/etc/rc.d/ezjail.sh stop myjail
ezjail-admin delete -w myjail

Then, to make a new flavour and make a jail using that flavour, something like

cd /data/jails/flavours
cp -pr default myflavour
cd myflavour/etc
cp -p /etc/resolv.conf .
cp -p /etc/localtime .
ezjail-admin create -f myflavour myjail
/usr/local/etc/rc.d/ezjail.sh start myjail

At this point, you've created a new jail with your customizations, and would use jls again to find the jail-id, and jexec to start a shell inside the running jail.

A flavour may also contain packages you wish to install upon jail creation, and commands to execute when the jail is created. Check out the ezjail.flavour file in your flavour directory. I've used it to install common useful things like bash, vim, gmake, and libiconv and gettext which take a long time to build that you don't want to repeat for every jail.

Viewing a man file

This is one of those little things that I just want to jot down for myself so I have it written down until I learn it for good. To view a man file, that's not installed in the regular man file locations, just run

nroff -man filename | more

Stupidly simple, but unfortunately not mentioned in the manpage for man.

Returned from PyCon

Got back from PyCon 2006, in mostly one piece. Picked up a terrible cold at the conference, I suppose scrounging food off the same buffet tables as 400 other people wasn't the most hygenic thing in the world.

Attended the mainly web-oriented sessions, came away very impressed with Django. I had sort of blown it off before because I didn't like the look of the templating language, and the ORM seemed weird. But after seeing what's coming in the Removing the Magic branch, I think it will be much much nicer. Was even inspired to spend the little time I had there Monday morning and afternoon sprinting with the Django guys, but I don't see how one can sprint effectively in such a short time with the limited knowledge of the codebase I had. Maybe if I go next year ... and I know more Django ... and can spend more than a day there, then I could accomplish something useful during the time.

The TurboGears guys demonstrated some nice things with AJAX widgets, but the SQLObject part of TG has given me trouble in the past when working with an existing DB, and seems to get in the way more than it helps. Even so, the TG guys, and Ian Bicking seemed pretty cool, so I hope they polish things up a bit more. Maybe SQLObject 2 will be the answer, or maybe a switch to SQLAlchemy (which wasn't represented at the conference), would make TG a nicer environment to work in.

I've been struggling with Zope for some years now. From a user standpoint I guess it's OK, from a programmer standpoint it's a nightmare, both 2.x and 3.x. The documentation and community attitude have rubbed me wrong for a long time. I attended a couple Zope sessions at the conference, but didn't hear anything to inspire me to keep up with it. I'll probably switch what little Zope things I have going to Django/TurboGears/CherryPy/whatever.

The PyParsing presentation on writing an adventure game was interesting, wish I could have attended the more in-depth one but it conflicted with a Django session. PyParsing looks to make a hard job pretty easy, and I'd love to play with it somewhere.

The party at NerdBooks had some decent food, they had a pretty deep selection of books, and the prices on some of the things I looked up were much better than Amazon. Will definitely look there next time I need something.

Lastly, I hope Django or someone who was at the sprint uses the codename "Vacuum Assassin" somewhere. That would just be too cool.

Self-Signed SSL Certificates

Quite often I find myself needing to generate self-signed certificates for use with OpenSSL. There are only three steps required...

Generate a key file, named ssl.key for example:

openssl genrsa -out ssl.key 1024

Generate a Certificate Signing Request for the key, named ssl.csr in this example. You'll be asked a bunch of questions, when asked for Common Name (eg, YOUR name) be sure to enter the domain-name you're making the certificate for (such as www.foobar.edu).

openssl req -new -key ssl.key -out ssl.csr

Generate a signed certificate given the request and key, valid for 10 years (3650 days) and named ssl.crt in this example. When you're done, the ssl.key and ssl.crt files are what you usually need to install in your server.

openssl x509 -req -days 3650 -in ssl.csr -signkey ssl.key -out ssl.crt

As a bonus, here's how to view the contents of a certificate file named ssl.crt

openssl x509 -in ssl.crt -text

mod_python segfault fixed

Just as a followup, it seems the segfault in mod_python on FreeBSD I mentioned before was found and fixed. Turns out to not be any kind of pointer/memory corruption like I thought, but rather a mishandled return code from an APR (Apache Portable Runtime) function. Oh well, I got to play with gdb, ddd, and valgrind a bit, which is good stuff to be familiar with.

Restoring Boot Sectors in FreeBSD

At work the other day, we had a long power outage, and afterwards one of our FreeBSD 5.2.1 boxes refused to come back up. It'd power up, go through the BIOS stuff, show the FreeBSD boot manager that lets you select which slice to boot, but when you hit F1, the screen would go black and the machine would reset.

Booted off the 5.2.1 install CD, and after entering fixit mode, was able to mount the disk and see that the files seemed to be intact. Couldn't run fsck though, the 5.2.1 CD seemed to be missing fsck_4.2bsd.

FreeSBIE 1.1 on the other hand, was able to fsck the disk, but that didn't solve the problem. Next guess was that something in the /boot directory was hosed. I'd setup the machine to do weekly dumps of the root partition to another machine, and was able to extract /boot from a few days before and pull it back onto this machine over the network using FreeSBIE, but it still wouldn't boot.

Next theory was that something in the boot sectors was bad. First tried restoring the MBR (Master Boot Record) from copy that's kept in /boot - even though it was working well enough to show the F1 prompt to select the slice. Wanted to keep what 5.2.1 had been using, so mounted the non-booting disk readonly and made sure to have boot0cfg use the copy there instead of anything that might have been on the FreeSBIE disc.

mkdir /foo
mount -r /dev/twed0s1a /foo
boot0cfg -B -b /foo/boot/boot0 /dev/twed0

Unfortunately, that didn't help. Each slice (partition in non-BSD terminology) also has boot sectors, and to restore them, turns out you use the bsdlabel (a.k.a. disklabel) utility. Again from FreeSBIE:

mkdir /foo
mount -r /dev/twed0s1a /foo
bsdlabel -B -b /foo/boot/boot /dev/twed0s1

That did it. Apparently something in the slice's boot sectors was messed up.

Getting rid of ugly fonts in Firefox

Lately I've been using Firefox on DragonFlyBSD with xorg installed from pkgsrc, and one thing that bugged me was that when reading Advogato, the fonts on that page looked like crap. The CSS stylesheet shows "lucida" as the preferred font, and my machine evidently was using a bitmap font for that.

At first I thought, just get rid of the bitmapped fonts from the FontPaths listed in /etc/X11/xorg.conf, but surprisingly that didn't seem to have any effect, at least on Firefox.

Secondly, I tried just removing those bitmap font directories completely, such as /usr/pkg/xorg/lib/X11/fonts/75dpi/ and that did work, but seemed a little clumsy in that an update to xorg would probably replace them.

Finally, stumbled across Fontconfig's files, and saw that there is a whole separate configuration of font paths and such, starting in /usr/pkg/etc/fontconfig/, which explains why changing the xorg.conf FontPath didn't work. Turns out there are even some optional configs in /usr/pkg/etc/fontconfig/conf.d/ including a no-bitmaps.conf which will cause fontconfig to "blacklist" the bitmap fonts.

The Fontconfig user manual mentions that things in conf.d/ are processed if they begin with decimal digits. So to enable that no-bitmaps.conf, I just made a symlink.

cd /usr/pkg/etc/fontconfig/conf.d
ln -s no-bitmaps.conf 10barryp-no-bitmaps.conf

Then, just had to stop/restart Firefox to see the results.

It would be nice to be a bit more selective about what gets blacklisted, so that non-Roman characters not supported in the scalable fonts on my machine would have some chance of displaying. I'll have to work on that.