bpgsql 2.0 alpha 1

For many years I've been using bpgsql, my own pure-Python PostgreSQL client, and I've finally sat down and got things somewhat polished up enough to put together as a real package.

One thing that motivated the work was the desire to use in with Django - after seeing psycopg2 do some funny things when used under mod_wsgi. There's no doubt it's slower, but it's much easier to hack on, and might be of interest to people running Djano under other Pythons such as PyPy or Jython. Getting it to pass all the Django unittests really ironed out a lot of bugs, so I think it's in fairly decent shape now.

RabbitMQ FreeBSD port

I was happy to see a FreeBSD port added for RabbitMQ, net/rabbitmq, although I found a couple problems with it: it doesn't start automatically when your machine or jail boots, and when building the rabbitmq-erlang-client, it errors out with:

src/amqp_channel.erl:28: can't find include lib "rabbitmq_server/include/rabbit.hrl"
src/amqp_channel.erl:29: can't find include lib "rabbitmq_server/include/rabbit_framing.hrl"

I worked on the port a bit, and submitted a bug report and patch, ports/127033, that fixes these problems.

amqplib 0.5

Put out a new release of py-amqplib, labeled 0.5, featuring the reworking mentioned earlier of how frames from the server are handled, and a big speed-improvement in receiving messages that was prompted by doing some profiling after reading Initial Queuing Experiments on the Second p0st blog.

Erlang-inspired improvements to amqplib

Programming Erlang cover I've been reading Joe Armstrong's Programming Erlang book, and it's been a real eye-opener. I've been intrigued by functional programming languages like Haskell and OCaml for some time, but it's been hard seeing how they'd be used for real programs, instead of just silly little factorial or quicksort examples. Joe's book is awfully well written and gives lot of very clear examples, so now when I look at bits of code like RabbitMQ, it doesn't seem like a bunch of gibberish :)

Haven't finished the book yet, but one thing I picked up from it already was the interesting way Erlang handles messages sent to processes, in how they're pattern-matched and saved for later if they're not what the process is looking for right at that moment.

At the heart of amqplib versions 0.3 or lower is a terrible mess that tries to deal with waiting for particular AMQP frames. Previously it would raise exceptions in some situations when it really shouldn't have. Specifically: if you had called basic_consume on a channel, and then called some other synchronous method like another basic_consume call - while it was expecting a basic_consume_ok response a basic_deliver could arrive and the library would raise an Exception because that wasn't expected.

A lame workaround is to use the nowait option most calls have, but I've now reworked things, cleaning up a lot of ugly code, and saving unexpected messages for later, similar to how Erlang does it. So now I believe the client library behaves in the way you'd generally expect. The improved code is currently in the Mercurial repository, and will be put out as a new release after it's had a chance to settle.

Logitech QuickCam Communicate STX

Earlier, I wrote about setting up a Mac Mini user running OSX Tiger 10.4.x with a Logitech QuickCam Pro 5000, and how there was trouble using the camera's microphone. Fixing it involved mucking around with Mac OSX kernel extensions, and later automatic updates seem to have messed that up.

Rather than keep fighting with OSX, I swapped out that webcam for a Logitech QuickCam Communicate STX. Specifically, the model with part # 961464-0403.

It's not supported by default with OSX 10.4.x, but it is supported by macam, which is ridiculously easy to install. After doing so it worked fine with Skype, and the auto-exposure feature even worked, where it adjusted itself automatically for the lighting in the room.

SSH in a FreeBSD jail

I've been running lots of FreeBSD jails on various servers I maintain, and one thing I've noticed is that using ssh or scp from inside a jail often results in the error: Host key verification failed. A little Google searching turns up this explanation, that the problem is caused when you jexec into the jail instead of logging in normally through SSH.

I often run the jails in a pretty minimal way and don't really want to run sshd in them, and fortunately the problem can be worked around somewhat. Apparently the Host key verification failed. error is caused when SSH is unable to show you this type of prompt:

The authenticity of host 'foobar.edu' can't be established.
DSA key fingerprint is 7c:ac:b0:da:be:3c:c2:00:00:00:00:ce:db:fb:49:77.
Are you sure you want to continue connecting (yes/no)?

when connecting to a host you haven't connected to before. All you have to do to get around this is manually add a line to the jail's ~/.ssh/known_hosts for the server you're trying to connect to, probably by copying one from a known_hosts on another box or outside the jail.

Once past that, you may find that SSH is still unhappy in the jail if you don't have publickey authentication setup with the server you're trying to connect to, with an error like:

Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).

Fixing that is just a matter of generating or copying a private/public key pair into the jail's ~/.ssh directory, and putting the public key on the server you're connecting to.

Mac OSX USB-Serial Adapter

At work I occasionally need to work with devices that have serial interfaces, like Cisco Access points, and wanted to do so with my MacBook. One particular USB-to-Serial adapter that I found works OK is a Sabrent SBT-USC1K, which uses the same MacOSX Prolific driver I had installed a while back for use with a GPS. I've used this on both MacOSX Tiger (10.4) and Leopard (10.5).

There are probably lots of USB-Serial adapters that work with a Mac, and even use that same driver. I just thought I'd jot down one in particular, so someone doesn't have to make a wild guess and order something hoping it happens to work with OSX.

"man" annoyance

I work on several different Unix-type machines, some FreeBSD, some Linux, and there are always small differences between them that can be very annoying.

One that's driven me crazy many times is how on some machines, when using man to read a manpage, after getting to the part I'm interested in, and hitting 'q' to quit - the contents of the manpage completely disappear and the screen is restored to what was shown before I ran man. Other machines I work on don't do that - the contents of the manpage remain on the screen so you can see them as you type the next command.

After finally getting fed up, I looked into this and it turns out the machines that were acting the way I like have the PAGER environment variable set to more, so man uses that instead of less which has the screen-restore-on-quit behavior. Adding:

PAGER=more; export PAGER

to .bashrc seems to have done the trick on my Ubuntu box. Seems the FreeBSDs already had this by default.

It's not a big deal, but I'm just jotting this down in case anyone else has the same annoyance. Good luck finding this in Google though, if you're just using words like 'more', 'less', and 'man' ;)

Daemontools mishap

While working on my DiskCompare.com website I had a weird problem with daemontools. The Django process I had running under daemontools became unresponsive, it wouldn't shutdown normally with svc -d, and after kill -9 and restarting, it still wouldn't respond - it just seemed hung up or frozen. No messages in /var/log/messages, but running the service manually outside of daemontools worked fine. What the heck?

I had the service also setup to run multilog, and it turns out that I had mistakenly changed the ownership of the log directory it was supposed to be writing to. My guess is that it was the multilog process that was really hung up, since it couldn't write to the files it wanted to, and that my Django processes were then blocked because the pipe between them and multilog was full.

Simply correcting the permissions of the log directory cleared the jam and things took off again. So if you're seeing strange behavior with daemontools, that's one area to check out.

DiskCompare.com

What's a good deal in hard disks? I don't mean which reseller sells a particular model for less than other resellers - but how does one model compare to another in terms of what you get for the money. If you're looking to buy a 400gb disk, does it make sense to spend a little more for a 500gb disk? or are you paying an excessive premium for that extra 100gb?

A good way to compare disks is to look at their value in terms of Gigabytes-per-dollar. Resellers usually don't display that information or make it easy to compare one disk against another, so I whipped together a website named DiskCompare.com that does that, based on prices from Newegg.

So for example, if you're interested in a Seagate 7200.10 500gb disk, you'd see a page like the one below. It compares the Seagate to other 500gb drives, and possibly to larger drives that are not too far off in price - and shows that some Western Digital models give you more for the money (at least at the moment I took this screenshot), although one is at the expense of a smaller cache.

screengrab of dicksompare.com

Other things it shows are relevant reviews, recent Newegg price changes, and price histories for each drive (some of those price changes are a bit shocking, today one SAS drive went up $390)

The site is still a work-in-progress, powered by Django, Python, PostgreSQL, and jQuery. It's been an interesting winter-time project, something to do when it's too dang cold to be outside. If you happen to follow the links on the site to Newegg and make a purchase, I'll get a small commission to help defray my bandwith costs, although I hope the site isn't too obnoxious about it.