The Little Hacker's Room

The distribuTorr project is coming along quite nicely, within a few months, there should be a code release and my hope is that some volunteers will try it out.

After some consideration, I considered XMLRPC over HTTP to be inadequate for my purposes. The main issues with XMLRPC over HTTP are binary bulk transfers (which would be base64 encoded, inefficient/waste of bandwidth), and that the xmlrpc-c library provides poor pipelining, even with the help of cURL as a transport backend.

I considered coding my own protocol, but I felt this would be inappropriate and would waste considerable time. So I started searching for a message encapsulation and encoding protocol. With a quick google search, I found GMime. This library encodes messages using MIME, the same mechanism used in email, for attachements, OpenPGP, S/MIME and HTML to name a few. Best of all, the library comes with OpenPGP support, the authentication mechanism that is planned for distribuTorr. The XMLRPC requests will now be carried over MIME, possibly accompanied by other chunks of data and potentially signed and encrypted much the same way OpenPGP email is.

By default, GMime only supports a GZip compression filter. GZip can be appropriate for many uses, it is a good compromise of speed and compression ratio. Wishing to save some bandwidth and latency in distribuTorr, I decided to code an LZO compression filter for GMime. LZO is a lightning fast compression algorithm that gives a pretty good compression ratio. It is ideal for network compression.

Within a few days of spare time, I have managed to code gmime-filter-lzo. My tests show that it is as fast as expected and seems quite robust for software in this stage of development. This code is released under the GPL, feel free to use it for your own needs. You may also wish to read the README file or consult the git repository.