User login

Nathan Overall's blog

30

Oct

2012

Well the year is pretty much over now. Just submitting the last few assignments.

520 Report hand in went well. I gauged the time well and happily submitted on time. Although my word count was lower than everyone else I feel the reduction in words was made up by a stronger focus and keeping my arguments concrete. I guess I'll find out if that was wise in due course.

It's been an awesome year being part of WAND and everything that entails. Thanks to everyone involved and good luck in the future.

21

Oct

2012

Been working pretty much constantly on my report for the past few weeks now. At this point I have a bit of tidying up before I write my conclusion and hand it in on tuesday. Overall I'm pretty happy with the report. I've received really good feedback from Brendon and Scott regarding how to improve and clean up my writing. At this point I'm on schedule to finish on time.

01

Oct

2012

Pretty full on with assignments this week and starting to work through writing my report. I hacked up a webpage, with a little help from Joe, so people can see our progress http://wand.net.nz/~no15/index.html . Hoping to get most of the WAND 520 students on there too.

Had a play with pypy (JIT Complier for python). Should be handy for getting more performance out of my collector. Also thought about creating an importer specifically for bulk importing data as running the collector can be slow for imports of existing data.

19

Sep

2012

Another busy week with Assignments. It's getting to that crunch time of the year, and what with thhe 520 final write up I suspect it's going to be a pretty hard slog.

Attended the HPC Roadshow on Thursday along with Brad, Brendon, and Shane. We met up with Jamie there too. Some interesting talks and it's good to build knowledge about subjects outside my usual domain.

09

Sep

2012

Had my honours presentation this week. Overall I was pretty happy with how things went. I felt The feedback I received during my practice talk made a major difference to the final presentation. I'd like to thank Richard (my supervisor), Scott (from Lightwire), Tony, Brendon, Shane, and all the other WAND people who helped critique my original presentation.

Looking at my 520 now I need to find some time to finish up the final major component, the connector. Then testing and benchmarking to see what I need to do performance wise. Making some rough notes for the final report is probably not a bad idea either.

05

Sep

2012

Spent last week preparing for my 520 presentation. Practice talks went well.

29

Aug

2012

Last week wasn't very productive. I was mostly tied up with lectures and revising. I did give Shane my project to start using. He quickly got the hang of how things worked and wrote a module to push data into a database. Hopefully I'll find some time to do some initial benchmarking on reading the data to see how things actually perform.

20

Aug

2012

Didn't get any 520 work done last week. Pretty flat out playing catch up at the moment after my time off. Slowly getting back on top of assignments and lectures.

12

Aug

2012

Haven't done a weekly report in a while due to being sick. Luckily I kept working on my 520 so I'm not horribly behind.

Firstly I made some pretty dramatic changes to the main code structure to improve performance. This seems to have paid off, making improvements to not only the performance but also the flow of the internal code.

All the database implementation is done now. I've made all the module code as agnostic as possibly so any module can be loaded without changes to the main code base. In the end I used sqlalchemy for the database abstraction layer. The framework is incredibly powerful and makes it easy to add other features at a later date.

The forwarding of the data had to be re-engineered when I restructured the main code so this needs to be re-implemented. Hopefully that's not going to be too much of a task at this point.

Due to being sick I have a fair amount of University work to catch up on so I'm not sure how much I'll be able to get done next week.

21

Jul

2012

Started pulling things together over the past few weeks. The modules are now all integrated into the main application. Some of the forking and threading code needs some TLC but things are at least functional at this point. The main problem is scalability. I'm starting a thread for each data source which falls over when you follow a few hundred data sources. I need to be more clever about identifying resources and ensuring I'm not reading from them more than once for multiple streams. I may also require some changes/additions to the AMP api to allow me to request multiple data items in a single request.

Next Brendon wants me to stream the data (from multiple sources) in order. I'm just restructuring code so I can get at the time codes together, however I've been sick on and off this week which hasn't helped progress. After that I'll try and get the database implementation going.