I attended OSCON for the first time this year, and to celebrate I thought I'd wrap up the Python amqplib library a bit and consider it more-or-less finished for what it is (a simple blocking 0-8 client), and call it 1.0.0 You can find it on the in PyPi and Google Project Hosting

It's definitely a worthwhile upgrade in that it's significantly faster than amqplib 0.6.1, and has a fair number of bug fixes. Also noteworthy are support for Python 3.x (via 2to3) and IPv6

I've created Google Code projects for my amqplib and bpgsql packages, to take advantage of their nice infrastructure including issue tracking.

Wrapped up another release of py-amqplib, version 0.6 - which features a major reorganization of the codebase to make the library more maintainable and lays the groundwork for an optional thread-assisted mode that allows for flow control and timeouts (being worked on in a development repository).

Normally I try to avoid dealing with PHP if at all possible, but there is now a PHP port of py-amqplib called php-amqplib, and I offered to help out with it a bit. Maybe partially out of guilt for having written the mess of Python code it was based on :)

I thought it would be handy to work on it using my MacBook. OS X 10.5 (Leopard) has PHP 5.2.6 built in standard, but unfortunately it doesn't have the bcmath extension included, which php-amqplib makes use of. Turns out building the module wasn't that difficult. This page got me going - although building bcmath was much simpler. Since I had the Apple Developer Tools for 10.5 installed, it was just a matter of ...

• mkdir /SourceCache
• cd /SourceCache
• fetch https://www.php.net/get/php-5.2.6.tar.gz/from/this/mirror
• tar xzvf php-5.2.6.tar.gz
• cd php-5.2.6/ext/bcmath
• phpize
• ./configure
• make
• sudo make install
• cd /etc
• cp php.ini.default php.ini

And then edit /etc/php.ini to make these two changes:

--- php.ini.default     2008-07-15 14:19:15.000000000 -0500
+++ php.ini     2008-12-08 21:44:52.000000000 -0600
@@ -483,7 +483,7 @@
 user_dir =

 ; Directory in which the loadable extensions (modules) reside.
-extension_dir = "./"
+;extension_dir = "./"

 ; Whether or not to enable the dl() function.  The dl() function does NOT work
 ; properly in multithreaded servers, such as IIS or Zeus, and is automatically
@@ -595,6 +595,7 @@
 ; needs to go here.  Specify the location of the extension with the
 ; extension_dir directive above.

+extension=bcmath.so

 ; Windows Extensions
 ; Note that ODBC support is built in, so no dll is needed for it.

After that, I was able to run the amqp_test.php file for the first time, sending a message and receiving it in py-amqplib's demo/demo_receive.py

I was happy to see a FreeBSD port added for RabbitMQ, net/rabbitmq, although I found a couple problems with it: it doesn't start automatically when your machine or jail boots, and when building the rabbitmq-erlang-client, it errors out with:

src/amqp_channel.erl:28: can't find include lib "rabbitmq_server/include/rabbit.hrl"
src/amqp_channel.erl:29: can't find include lib "rabbitmq_server/include/rabbit_framing.hrl"

I worked on the port a bit, and submitted a bug report and patch, ports/127033, that fixes these problems.

Put out a new release of py-amqplib, labeled 0.5, featuring the reworking mentioned earlier of how frames from the server are handled, and a big speed-improvement in receiving messages that was prompted by doing some profiling after reading Initial Queuing Experiments on the Second p0st blog.

I've been reading Joe Armstrong's Programming Erlang book, and it's been a real eye-opener. I've been intrigued by functional programming languages like Haskell and OCaml for some time, but it's been hard seeing how they'd be used for real programs, instead of just silly little factorial or quicksort examples. Joe's book is awfully well written and gives lot of very clear examples, so now when I look at bits of code like RabbitMQ, it doesn't seem like a bunch of gibberish :)

Haven't finished the book yet, but one thing I picked up from it already was the interesting way Erlang handles messages sent to processes, in how they're pattern-matched and saved for later if they're not what the process is looking for right at that moment.

At the heart of amqplib versions 0.3 or lower is a terrible mess that tries to deal with waiting for particular AMQP frames. Previously it would raise exceptions in some situations when it really shouldn't have. Specifically: if you had called basic_consume on a channel, and then called some other synchronous method like another basic_consume call - while it was expecting a basic_consume_ok response a basic_deliver could arrive and the library would raise an Exception because that wasn't expected.

A lame workaround is to use the nowait option most calls have, but I've now reworked things, cleaning up a lot of ugly code, and saving unexpected messages for later, similar to how Erlang does it. So now I believe the client library behaves in the way you'd generally expect. The improved code is currently in the Mercurial repository, and will be put out as a new release after it's had a chance to settle.

I noticed the other day that my two RabbitMQ servers were consuming more and more memory - one had gone from an initial 22mb size to over 600mb. As I sat and watched it would grow by 4k or so at regular intervals.

I think what had happened is that I had created an exchange which received lots of messages, and then ran scripts that created automatically-named queues bound to that exchange, but defaulted to not auto-deleting them. I ran these scripts many many times, which left many many queues on the server, all swelling up with lots of messages that would never be consumed. Good thing I caught it, it might have eventually killed my server.

This message in the rabbitmq-discuss list gives useful info on how to get in and see what queues exist on a RabbitMQ server, and how big they are.

It seems to me that having the auto_delete parameter of Channel.queue_declare() default to False is a really bad idea. If you want to keep a queue around after your program exits, I think you should explicitly say so, so I changed the default to True. The Channel.exchange_declare() also has a auto_delete parameter, which I also change the default to True for consistency.

I also did some work on supporting the redirect feature of AMQP, where a server you connect to can tell you to go somewhere else, useful for balancing a cluster. I don't actually have a RabbitMQ cluster, so I put together a utility to fake an AMQP server that tells you to redirect. It works well enough to run the uniitests unchanged against it, each test case being redirected from the fake server to the real server.

With those two changes, I put out a 0.2 release, on my software page and on the Cheeseshop.

I broke down and put together a tarball of my Python AMQP library, and stuck it up as a release 0.1 on the software section of this website, under the section py-amqplib.

Interestingly, someone hit the page and downloaded the tarball less than 3 minutes after I dropped a note about it to the RabbitMQ discussion list - so I guess there's at least some interest out there in this sort of thing :)

For some time I've been using Spread as a messaging bus between processes and machines using Python bindings, but there are a few things that make it not quite ideal for what I've been trying to do.

• There's no access control
• Messages are non-persistent - so if a receiver daemon is down and some important message comes through, it's SOL
• The wire protocol is not documented, the docs basically just say use the C client library.
• The Python bindings to the C library have a glitch of some sort when used in py-exim-localscan, I had to resort to a small ctypes wrapper to get around this.

I ran across the Advanced Message Queuing Protocol(AMQP), with RabbitMQ as one implementation of the protocol, that looks like a better fit for my needs.

There's a Python client library available named QPID, but there are a few issues with that:

• Relies on threading, which is trouble when Python is embedded in something else, or if you want to try using it in Stackless Python
• Lacking documentation
• Has to load a big AMQP XML Spec file, which takes a few seconds.

I decided to take a whack at my own AMQP client, partially as a learning excercise to learn more about the protocol. I wrote a program to take the AMQP 0-8 spec file and statically generate the skeleton of a Python module, and then fleshed it out by hand. The generator is able to put lots of documentation from the spec file into Python docstrings, so the pydoc of this module is fairly decent. Because the module is statically generated, it should be easier to debug than QPID which generates lots of stuff on-the-fly. It's also much faster at making the first connection because it's not parsing the spec file. I also thew in SSL support in since it wasn't too difficult.

It has a ways to go, and some parts are probably naively conceived, but it does seem to work.

The first thing I've used it for is a syslog->AMQP bridge. I've setup my FreeBSD syslogd to feed all info or higher events to a Python daemon, which extracts the date, time, program name, host name, etc and reformats as an AMQP message and published to a 'syslog' topic exchange with the program name as the routing key.

My plan is then to write other daemons that subscribe to the 'sshd' topic for example, and then generate higher-level messages that say things like: 'block IP address xx.xx.xx.xx' in case of failed login attempts. Then i just need one daemon to listen for these firewall control message and make changes to the PF tables.

It's fun stuff. The only weak part is that there's no way to tell if the original syslog message was spoofed, but after that point, AMQP access controls should keep things trustworthy.

See py-amqplib for a Mercurial repository and eventual downloads.