User login

Weekly Report

01

Apr

2012

This week I got a lot more bogged down in finishing an assignment than I would have liked. I spent some time this weekend to a make up a little and made a huge point of progress in investigating messaging systems.

It turns out there are two main types of messaging systems - message queuing and publish-subscribe messaging (note the lack of the word 'queuing'). Message queuing is designed for distributing messages so each message is delivered to exactly one consumer (e.g. a work unit in in a distributed application). Publish-subscribe messaging is more what is wanted for this project - many publishers publishing to one or more subscribers.

Many messaging queuing system can do publish-subscribe too but there are some designed specifically for publish subscribe like Facebook Scribe, Apache Flume and Apache Kafka (developed by LinkedIn); designed for things like reliably aggregating logs. I've only had a glance over them so far, but Facebook Scribe at least appears to fulfil the requirement of durability and reliability on the node side.

I've also attached a copy of my proposal to this post, in case anyone is interested