Got back from PyCon 2006, in mostly one piece. Picked up a terrible cold at the conference, I suppose scrounging food off the same buffet tables as 400 other people wasn't the most hygenic thing in the world.

Attended the mainly web-oriented sessions, came away very impressed with Django. I had sort of blown it off before because I didn't like the look of the templating language, and the ORM seemed weird. But after seeing what's coming in the Removing the Magic branch, I think it will be much much nicer. Was even inspired to spend the little time I had there Monday morning and afternoon sprinting with the Django guys, but I don't see how one can sprint effectively in such a short time with the limited knowledge of the codebase I had. Maybe if I go next year ... and I know more Django ... and can spend more than a day there, then I could accomplish something useful during the time.

The TurboGears guys demonstrated some nice things with AJAX widgets, but the SQLObject part of TG has given me trouble in the past when working with an existing DB, and seems to get in the way more than it helps. Even so, the TG guys, and Ian Bicking seemed pretty cool, so I hope they polish things up a bit more. Maybe SQLObject 2 will be the answer, or maybe a switch to SQLAlchemy (which wasn't represented at the conference), would make TG a nicer environment to work in.

I've been struggling with Zope for some years now. From a user standpoint I guess it's OK, from a programmer standpoint it's a nightmare, both 2.x and 3.x. The documentation and community attitude have rubbed me wrong for a long time. I attended a couple Zope sessions at the conference, but didn't hear anything to inspire me to keep up with it. I'll probably switch what little Zope things I have going to Django/TurboGears/CherryPy/whatever.

The PyParsing presentation on writing an adventure game was interesting, wish I could have attended the more in-depth one but it conflicted with a Django session. PyParsing looks to make a hard job pretty easy, and I'd love to play with it somewhere.

The party at NerdBooks had some decent food, they had a pretty deep selection of books, and the prices on some of the things I looked up were much better than Amazon. Will definitely look there next time I need something.

Lastly, I hope Django or someone who was at the sprint uses the codename "Vacuum Assassin" somewhere. That would just be too cool.

Quite often I find myself needing to generate self-signed certificates for use with OpenSSL. There are only three steps required...

Generate a key file, named ssl.key for example:

openssl genrsa -out ssl.key 1024

Generate a Certificate Signing Request for the key, named ssl.csr in this example. You'll be asked a bunch of questions, when asked for Common Name (eg, YOUR name) be sure to enter the domain-name you're making the certificate for (such as www.foobar.edu).

openssl req -new -key ssl.key -out ssl.csr

Generate a signed certificate given the request and key, valid for 10 years (3650 days) and named ssl.crt in this example. When you're done, the ssl.key and ssl.crt files are what you usually need to install in your server.

openssl x509 -req -days 3650 -in ssl.csr -signkey ssl.key -out ssl.crt

As a bonus, here's how to view the contents of a certificate file named ssl.crt

openssl x509 -in ssl.crt -text

Just as a followup, it seems the segfault in mod_python on FreeBSD I mentioned before was found and fixed. Turns out to not be any kind of pointer/memory corruption like I thought, but rather a mishandled return code from an APR (Apache Portable Runtime) function. Oh well, I got to play with gdb, ddd, and valgrind a bit, which is good stuff to be familiar with.

At work the other day, we had a long power outage, and afterwards one of our FreeBSD 5.2.1 boxes refused to come back up. It'd power up, go through the BIOS stuff, show the FreeBSD boot manager that lets you select which slice to boot, but when you hit F1, the screen would go black and the machine would reset.

Booted off the 5.2.1 install CD, and after entering fixit mode, was able to mount the disk and see that the files seemed to be intact. Couldn't run fsck though, the 5.2.1 CD seemed to be missing fsck_4.2bsd.

FreeSBIE 1.1 on the other hand, was able to fsck the disk, but that didn't solve the problem. Next guess was that something in the /boot directory was hosed. I'd setup the machine to do weekly dumps of the root partition to another machine, and was able to extract /boot from a few days before and pull it back onto this machine over the network using FreeSBIE, but it still wouldn't boot.

Next theory was that something in the boot sectors was bad. First tried restoring the MBR (Master Boot Record) from copy that's kept in /boot - even though it was working well enough to show the F1 prompt to select the slice. Wanted to keep what 5.2.1 had been using, so mounted the non-booting disk readonly and made sure to have boot0cfg use the copy there instead of anything that might have been on the FreeSBIE disc.

mkdir /foo
mount -r /dev/twed0s1a /foo
boot0cfg -B -b /foo/boot/boot0 /dev/twed0
reboot

Unfortunately, that didn't help. Each slice (partition in non-BSD terminology) also has boot sectors, and to restore them, turns out you use the bsdlabel (a.k.a. disklabel) utility. Again from FreeSBIE:

mkdir /foo
mount -r /dev/twed0s1a /foo
bsdlabel -B -b /foo/boot/boot /dev/twed0s1
reboot

That did it. Apparently something in the slice's boot sectors was messed up.

Lately I've been using Firefox on DragonFlyBSD with xorg installed from pkgsrc, and one thing that bugged me was that when reading Advogato, the fonts on that page looked like crap. The CSS stylesheet shows "lucida" as the preferred font, and my machine evidently was using a bitmap font for that.

At first I thought, just get rid of the bitmapped fonts from the FontPaths listed in /etc/X11/xorg.conf, but surprisingly that didn't seem to have any effect, at least on Firefox.

Secondly, I tried just removing those bitmap font directories completely, such as /usr/pkg/xorg/lib/X11/fonts/75dpi/ and that did work, but seemed a little clumsy in that an update to xorg would probably replace them.

Finally, stumbled across Fontconfig's files, and saw that there is a whole separate configuration of font paths and such, starting in /usr/pkg/etc/fontconfig/, which explains why changing the xorg.conf FontPath didn't work. Turns out there are even some optional configs in /usr/pkg/etc/fontconfig/conf.d/ including a no-bitmaps.conf which will cause fontconfig to "blacklist" the bitmap fonts.

The Fontconfig user manual mentions that things in conf.d/ are processed if they begin with decimal digits. So to enable that no-bitmaps.conf, I just made a symlink.

cd /usr/pkg/etc/fontconfig/conf.d
ln -s no-bitmaps.conf 10barryp-no-bitmaps.conf

Then, just had to stop/restart Firefox to see the results.

It would be nice to be a bit more selective about what gets blacklisted, so that non-Roman characters not supported in the scalable fonts on my machine would have some chance of displaying. I'll have to work on that.

Other people have reported the same problem with mod_python on FreeBSD I had seen before, so I'm happy that I'm not losing my mind.

I took a stab at using Valgrind to find the problem. Didn't actually find anything, but I thought I'd jot down notes on how I went about this.

First, the Valgrind port didn't seem to work on FreeBSD 6.0. When I tried running it against the sample code in the Valgrind Quick Start guide, it didn't find anything wrong with it. Ended up finding a FreeBSD 5.4 machine, which did see the expected problem.

Next, I built the Apache 2.0.x port with: make WITH_THREADS=1 WITH_DEBUG=1, and then built mod_python which uses APXS and picks up the debug compile option from that.

Then, in the mod_python distribution, went into the test directory, and downloaded a Valgrind suppression file for Python, valgrind-python.supp, and in it uncommented the suppressions for PyObject_Free and PyObject_Realloc (otherwise the Valgrind output is full of stuff that is really OK). Then tweaked test/test.py around line 307 where it starts Apache, to insert

valgrind --tool=memcheck --logfile=/tmp/valgrind_httpd --suppressions=valgrind-python.supp

At the front of the cmd variable that's being composed to execute httpd.

Finally, ran python test.py, and then looked at /tmp/valgrind_httpd.pid#### to see the results.

A while ago I threw together this script to automatically create package files for all installed ports on a FreeBSD box. That way, if a portupgrade doesn't work out, you can delete the broken package, and pkg_add the backup.

Stick this in /usr/local/etc/periodic/daily, and the system will automatically bundle up copies of the installed software and stick them in /usr/local/packages if they don't already exist in there.

#!/bin/sh
#
# Make sure backups exist of all installed FreeBSD packages
#
# 2005-03-20 Barry Pederson <bp@barryp.org>
#

ARCHIVE="/usr/local/packages"

#
# Figure out which pkg_tools binaries to use
#
if [ -f /usr/local/sbin/pkg_info ]
then
    PKG_TOOLS="/usr/local/sbin"
else
    PKG_TOOLS="/usr/sbin"
fi

#
# Make sure backup directory exists
#
if [ ! -d $ARCHIVE ]
then
    mkdir $ARCHIVE
fi

cd $ARCHIVE

for p in `${PKG_TOOLS}/pkg_info -E "*"`
do
    if [ ! -f ${p}.tgz ]
    then
        ${PKG_TOOLS}/pkg_create -b ${p}
    fi
done

Took another whack at getting PyBlosxom/SCGI working with Lighttpd, this time with better success. (I'm still getting up-to-speed with Lighttpd). This is working with the exact same SCGI setup I was working on the other day.

To elaborate a bit, the setup I'm trying to achieve is to:

• Have the blog to be completely under "/blog/" in the URL namespace
• Not get it confused with anything else that begins with "/blog" such as "/blog2".
• Use "/blog/static/" URLs for serving static resources like CSS stylesheets and images off the disk (instead of running those requests through PyBlosxom's CGI code).

This is what I ended up with, seems to work fairly well, and I'm impressed with how Lighttpd makes it easy to put together a understandable configuration.

#
# External redirection to add a trailing "/" if exactly 
# "/blog" is requested
#
url.redirect = (
                "^/blog$" => "http://barryp.org:81/blog/",
               )

#
# The PyBlosxom Blog, lives under the "/blog/" url namespace
#
$HTTP["url"] =~ "^/blog/" {
    #
    # Static resources served from the disk
    #
    $HTTP["url"] =~ "^/blog/static/" {
        alias.url = ("/blog/static/" => "/data/blog/static/")
    }

    #
    # Everything non-static goes through SCGI
    #
    $HTTP["url"] !~ "^/blog/static/" {
        scgi.server = ( "/blog" => (
                                     (
                                     "host" => "127.0.0.1",
                                     "port" => 8040,
                                     "check-local" => "disable",
                                     )
                                   )
        )
    }
}

Ran across Mark Mayo's blog entry: FastCGI, SCGI, and Apache: Background and Future, which discusses exactly the things I've been struggling with this weekend. I have to agree that sticking an interpreter like Python directly into Apache is a lot of trouble. I've delved into Apache sourcecode, and the mass of macros and #ifdefs is enough to send you running away screaming. To try and graft Python onto that is just begging for trouble - and I've had some experience myself with grafting interpreters onto other things.

Running your webcode in separate processes just makes a lot of sense. You have much more freedom with choice of language and version of language. You can easily run things under different userids, chrooted, in jails/zones, on completely separate machines, completely separate OSes, maybe within virtual machines running different OSes on the same hardware.

Anyhow, thought I'd mention this because Mark's writeup made a lot of sense to me and I thought it was worth keeping a link to it.

While doing a bit more searching for daemontools info, found the djb way website, which has some nice writeups on daemontools and djbdns (which I also use a fair amount).