User login

Nathan Overall's blog

03

Jun

2012

This week I didn't manage any more on my 520. I was ill on Tuesday which pretty much put me out of action for a few days. So in the end I barely got anything done all week.

Last week was more productive however. I manage to get a daemon going which accepts connections from a unix domain socket. At this point it expects an ID and timestamp and then feeds back all data for the source from that timestamp onwards. It then leaves the connection open and sends any new data as it arrives. I also implemented a simple command line and a way to restart the modules without restarting the whole server which should avoid having to restart the whole daemon all the time.

21

May

2012

Actually starting to make some progress on my 520 again. Turns out that's what evenings are for.

Tidied up my rrd implementation and fixed a bug where some of the data was ignored due to me misinterpreting the timestamp somewhere along the way. Started dealing with returning the new data each time the program is run. At the moment I'm just putting the last timestamp I read and the filename into a text file.

I'm now working on my Interim Report. Any guidance or suggestions are most welcome.

15

May

2012

Another week of fairly full on assignments (OpenCL and Wireless Sensor Nodes) which prevented me making progress on my 520. Trying to make an effort to spend more time at uni next week.

08

May

2012

Didn't really get anything done last week. I've been working flat out with assignments. Let's hope next week goes better.

01

May

2012

Spent a fair bit of my time doing my assignment for 513 last week. I did get some starting code towards building the databases while house sitting for my parents at the end of the week.

23

Apr

2012

I made some serious progress last week. I've now got my input plugins completed. I had to ditch the library I was using for parsing the RRDs as it was returning data in a totally broken way and I didn't have time to fix it. Instead I wrote a simple parser of the xml from rrdtool dump which seems to work nicely. I also wrote up an input plugin for amp which had a really nice API to work from. I started learning what I need to do for the database side of things. I had a chat to Christopher as has done some investigation into this area already and I may be able to incorporate some of his work into the project.

17

Apr

2012

Sadly I didn't get much done last week. I spent most of the time finishing other assignments and catching up on various things. I did have a look at how I was going to store the data in the database, particularly in respect to datatypes and table layouts. I'm hoping to work on my 520 full time this week.

10

Apr

2012

Finally wrote a script to pull apart an RRD. I've tried to be as flexible as possible. One thing that is going to be interesting is working out what schema the program inserting the data into the RRD is using. Column titles for the few RRDs I've pulled apart so far go from difficult to decipher to just plain useless (col1,col2,etc).

I also had a play with listening on a socket and passing the data over it. Next task on the list is to come up with a nice flexible protocol for transferring the data to the server.

03

Apr

2012

Ended up spending way more time than I should have on an assignment last week.

I did however spend some time researching different storage options and read an interesting paper on a storage solution google invented called dremel.

Annoyingly I have slowly fallen behind schedule. I will need to commit a large amount of time over the holidays to get back on track with my project.

26

Mar

2012

Spent most of last week writing my proposal which you can either find here:
http://goo.gl/fA1QI
or attached to this post. I managed to find quite a few documents commenting on the lack of scalability of RRD Tool and one from the original designer stating it's actual use case which was pretty helpful.

I did also get some time last week to learn how RRDs work (with some help from Brendon). I can now use python to pull apart RRDs. Sadly the documentation around pulling raw data out of RRDs is somewhat scarce so I'm having to work it out a lot as I go. Hopefully by next week I will have a nice script to pull values from Smokeping RRDs.