User login

Search Projects

Project Members

Message Queuing for Network Monitoring

Network monitoring systems, such as the AMP project, are more useful when measurements are taken from a diverse range of locations in order to provide information from different parts of networks. Modern message queuing protocols, such as those based on AMQP (Advanced Message Queuing Protocol), allow for more flexible and potentially interoperable message routing and more sophisticated test co-ordination than those currently used by AMP.

This project is to investigate message queuing protocols and their reliability and scalability in wide area networks, particularly with regards to coping when networks may not be behaving correctly (where network monitoring is most useful); and to recommend a solution for network monitoring systems. The chosen system will then be integrated into AMP to replace the existing simple message queuing and test co-ordination systems, for increased flexibility.

04

Jun

2012

Had a big assignment due this week, when it was done I was all ready to work on my project. Then I decided to reboot back into windows...and got bombarded with checkdisk errors - fun! It decided it might be fun to truncate some files without saying anything more than a file number and those are really annoying to translate. Nothing important seems to have been corrupted (against the copy on my laptop) and after several annoyingly long hardware checks I think it must have just been a fluke or a badly plugged in SATA cable - good way to use up the weekend. I've had this problem before - hopefully it isn't the Linux NTFS drivers (I have them set quite deliberately to norecover) or something, would be really nice to find what is causing it once and for all. It would also be nice if windows actually REPORTED the trickle of filesystem problem events it's logged over the last few weeks. Thankfully I have daily online backups that are versioned. Hopefully I can hold the computer together for a couple more weeks as there seem to be a couple of corrupted system files.

Interim report due on Friday - think I might use Google Docs for drafting that or at least save in my dropbox folder just in case.

Edit: My computer didn't wake properly this morning so I powered it off. Now the power button doesn't do anything - at least now I know it is a hardware problem. Looks like my suspicions about it being the power supply might be right afterall (the oldest component).

27

May

2012

Got less time that I would have liked to work on my project, as usual. Had a chat with Brendon about good C coding practices, string parsing and SSL. Between that, some SSL tutorials and wading through the OpenSSL documentation I managed to get a BIO socket working both with and without SSL transparently. This worked with HTTPS, but ActiveMQ wouldn't accept SSL STOMP connections without certificates/keys being set up (or I'd configured it wrong). Looked in to how to get that organised but it's a little convoluted with differing keystore types.

Managed to get my little toy app to send a STOMP CONNECT frame and get a reply so that's a start. Took me a couple of minutes to realize that I can't put a null at the end of null-terminated string with BIO_puts(). BIO also has a BIO_gets() which (usually) reads lines from the socket, which could be handy for header parsing as an alternative to tokenizing. In my OpenSSL documentation wading I came across that you can have authentication without encryption which I didn't know, and that BIO has nice support for non-blocking IO. In general OpenSSL doesn't seem too bad, just badly documented.

Next step is to figure out certificates, keys and the like; and get that working. Then I can start writing something resembling a STOMP library.

20

May

2012

Made some concrete decisions in the meeting on Monday: implementing a STOMP 1.1 C client, then dealing with persistence, then working on integrating into AMP. Was going to take another look at MQTT, but quickly noticed that unlike the Java version, the C version of the Eclipse Paho client (with built in persistence, the main advantage) doesn't do SSL and is a reasonably complicated library.

Thought implementing a STOMP library would be relatively straightforward, since it is a simple text-based protocol and remembering doing a simple HTTP server in a previous course. Of course that was in Java, and everything in C (particularly to do with strings and potential buffer overflows) is a lot more complicated. OpenSSL also has rather sparse documentation. There is a very lightweight C library, libstomp, that allows basic frame sending and receiving (using APR with hash tables for headers and the like), but it doesn't support SSL. Spent several hours trying to get my head around APR, but despite supporting OpenSSL crypto functions it doesn't actually seem to support SSL sockets.

Also had a quick look at Berkeley DB for persistence (since I came across support for it in APR), was a little excited about the 'recno' storage option (one of several) that is flat record number indexed, but it needs a record separating character which isn't going to work with binary messages.

06

May

2012

Had very little time this week with an assignment but I've pretty much decided on using a STOMP compatible custom system. This week I've been checking the requirements like failover possibilities are met (or can be made to meet). It would be nice if STOMP had brokerless support, but since it is a simple protocol I don't see too much of a problem with sending messages (as opposed to 'send') directly to nodes - I think it is better than using a totally custom protocol. There are some slightly irritating limitations with not being able to get message ID's at the publish side but these can be worked around with many STOMP servers supporting JMS-style header extensions (and any server just pass custom headers along anyway).

This would mean AMP could interoperate with any STOMP server (ActiveMQ. RabbitMQ, HornetQ) for sending results. Still need to double check and satisfy myself there are solid reasons for rejecting MQTT/ZeroMQ/Facebook Scribe. I feel AMQP's many incompatible protocol versions basically rule it out, even though it is a little more flexible than STOMP. Hope to start having a play with toy implementations next week. One of the things I want to figure out is how acks get passed around and how they relate to receipts as that could be an issue.

30

Apr

2012

Had a meeting with Richard and Brendon on Monday. It ended up being a fairly long discussion given I hadn't made up my mind too much. The plausible choices seem to be: running an AMQP broker in federation on each node, improving the existing system; or possibly MQTT but authentication is an issue. So, this week I looked at the cost of an AMQP broker - RabbitMQ needs around 40MB of Erlang VM, Qpid is around 10MB total including the client libraries. I also looked at exactly how federation works with Qpid and it seems feasible, but I'm worried about it using a different version of AMQP than everyone else (0.10) - losing most of the benefits of AMQP anyway (which are moot given the standard is changing).

The main issue with the current system (apart from being custom) was identified in the meeting to be unreliability in the client side persistence, so I've started having a look at libraries for that that are known reliable (the main choices seem to be embedded databases). A good option may be to integrate that and use something like the STOMP protocol for ease of interoperability (and possibly a 3rd party STOMP broker).

MQTT (designed for low power) authentication built in to the protocol is indeed plain text and only on connection so not at all useful, but I did find an ActiveMQ Apollo MQTT plugin that does support SSL; and confirmed the main open source MQTT client has built-in client-side persistence.

23

Apr

2012

Between spending a 4 day weekend at home and at my dad's work, coming back to a break-in at our flat (thankfully they only took one of my monitors and a hard drive and that's about it) and going to a funeral this weekend; I haven't got nearly as much research done as I would have liked.

I did manage to eliminate almost all of the less well known options (as most only do one-of-many message queuing, buried somewhere in the documentation) and almost finished figuring out the architecture (and mostly non-applicability due to everything being on the broker) of AMQP. I also planned to get assignments under way which didn't happen, so I doubt I'll have much time this week. Hopefully I should be able to get through re-checking the last couple of systems and collect my thoughts in the morning for the meeting with Richard and Brendon in the afternoon.

16

Apr

2012

Went back through a few of the messaging systems in more detail and got a much clearer overall picture. It seems there is some interesting politics around AMQP as one of the original developers (and developer of OpenAMQ), iMatix, abandonded it as poorly designed and getting nowhere in late 2010 and went on to develop ZeroMQ. The main implementation of RestMS was also part of OpenAMQ and also appears to be dead as there has been no activity on it's website or mailing list since then. Looked in to protobuf and thrift (as part of looking at Facebook Scribe) and they do seem like a great way to better define the message contents. There is definitely no perfect system so a good option may be to generalise the existing one.

Had a meeting with Richard to discuss progress and possibilities. We decided that this week I will finish examining the options in time for a meeting next Monday with Richard and Brendon to discuss the top candidates and the existing system; and hopefully make something resembling a decision.

08

Apr

2012

Spent several hours doing research earlier in the week. Found that Kafka and Flume are heavily java based which was disappointing. Haven't looked too much into Facebook Scribe yet. Knowing exactly what terms to look for now in other messaging systems has been helpful. RestMS seems interesting as it is designed by one of the developers of AMQP to address some of its limitations, and has queuing at the client side. There is definitely a lot of cross-compatibility between different messaging systems and protocols (e.g. RestMS and AMQP) which makes things a little easier.

Feeling like I'm losing the big picture a bit by spending an hour or two at a time between other things. This week I intend to spend more time in one go (due to the teaching recess) and go back through the options in detail, keeping track of requirements and features rather than the loose collection of notes I have so far.

01

Apr

2012

This week I got a lot more bogged down in finishing an assignment than I would have liked. I spent some time this weekend to a make up a little and made a huge point of progress in investigating messaging systems.

It turns out there are two main types of messaging systems - message queuing and publish-subscribe messaging (note the lack of the word 'queuing'). Message queuing is designed for distributing messages so each message is delivered to exactly one consumer (e.g. a work unit in in a distributed application). Publish-subscribe messaging is more what is wanted for this project - many publishers publishing to one or more subscribers.

Many messaging queuing system can do publish-subscribe too but there are some designed specifically for publish subscribe like Facebook Scribe, Apache Flume and Apache Kafka (developed by LinkedIn); designed for things like reliably aggregating logs. I've only had a glance over them so far, but Facebook Scribe at least appears to fulfil the requirement of durability and reliability on the node side.

I've also attached a copy of my proposal to this post, in case anyone is interested

25

Mar

2012

This week I spent most of my time writing my proposal and doing various assignments. While looking for background references I found quite a few useful looking papers on message queuing. Message queuing systems that are designed to queue on the server are definitely going to be an issue with AMQP based system, particularly since AMQP 0.9 doesn't specify communication in server clusters. Along with other options, the new version 1.0 being standardised (that isn't backwards compatible) might be worth looking at as it's designed to address some of the issues.

18

Mar

2012

This week I brainstormed requirements for the message queuing system from what I knew about AMP and had a meeting with Brendon to discuss additional requirements. The meeting was very helpful as Brendon also explained the current architecture and showed me how to find my way around the relevant parts of the AMP source code. The current message queuing system is rather simple (keep a secure connection open to server(s), queue messages on disk, simple acknowledgements, if connection dies try again later) so almost any dedicated message queuing system should be at least as flexible.

I also began compiling a list of possible message queuing systems (hardly any of which are listed on Wikipedia), and began tentatively eliminating some that don't meet basic requirements (e.g. being Java only) to narrow down the list to a more manageable size to more comprehensively compare.

11

Mar

2012

Hi! I'll be working on a more flexible message queuing system for AMP as my 520 honours project this year. This week I joined the WAND group, had an orientation meeting and a meeting with Richard Nelson, my supervisor, to discuss the beginnings of requirements and evaluation criteria (most importantly that it copes well with misbehaving networks).

One potential issue we discussed was that the default client/server centralised nature of AMQP (or at least RabbitMQ from my very brief look so far) isn't very conducive to reliability under adverse conditions. More advanced configurations (clusters of servers and the like) appear to be possible and will need investigating; as well as where queues actually reside (they appear to be stored on the server), which could complicate co-ordination. Casting a wider net than the most popular systems might be needed to find the most suitable solution to integrate (or even implement if none of them are, which could be interesting).