User login

Blogs

18

Jul

2011

Continued processing Waikato traces and adding more graphs to http://www.wand.net.nz/~salcock/longitude/ . Had a few segfault-related hiccups along the way which has slowed things down quite a bit.

Worked on a new analysis test that would divide traffic by source and destination AS using MIBs downloaded from routeviews. Turned out that libwandbgp couldn't really support maintaining a BGP route table from routeviews data for any length of time, so had to re-write a decent chunk of the library to be able to do so. Still not quite working yet, but starting to get there.

Received some patches for both libtrace and maji from one of our users, which I checked and integrated back into our software for future release.

15

Jul

2011

Started classes again this week - COMP518 and COMP514.

I fixed up my broken force directed layout type for the network map and started looking at the most adaptable way of allowing multiple layers in a graph.

Have started putting together a few slides for my talk next week to the class about my 520 project progress.

12

Jul

2011

Picked up working on SMTP state machines again this week. Decided the
first new step was to expand the flows I was learning from to include
those that didn't progress to the point of sending data (when testing
against a new dataset it was incorrectly dealing with flows truncated due
to greylisting etc). Unfortunately running it against that much extra data
meant it took longer than expected and had to be stopped for the upgrade
to spectre on Wednesday. While it was running I refactored a lot of the
code and spent time trying to make it both faster and more readable. After
the upgrade I also had to spend some time getting a recent version of R
running nicely again.

With the upgrade of relic I decided it was time to rewrite the emulation
network machine imaging and configuration control code to remove the
integration with ns2. It is now working again for imaging machines and can
configure basic networking for Linux hosts. Once I add in basic
configuration for FreeBSD/OpenBSD it will be at the same point as the old
system was. Hoping I can add in some more smarts and improve on the system
overall.

11

Jul

2011

Over the past week I added a couple more features to my network map such as an overview box and a debugging mode. The debugging mode just shows useful things like the frame rate, current scaling and translation offsets, and shows the outlines of nodes.
I think that I have now read and briefly summarised most of the relevant publications for my lit review.
Stayed at Hahei all weekend! Life of a student.

11

Jul

2011

Continued to have a few problems with processing large trace sets. Finally managed to get rid of all the memory leaks in my analysis code and fixed a tricky little libtrace bug that would cause processing to stop as soon as it hit a trace file less than 1 MB in size.

Started working on a web page to display all the graphs I'm creating nicely - http://www.wand.net.nz/~salcock/longitude/ . At the moment, only some of the Waikato I results are up, but will continue adding more results over the course of the week.

Tested and released new versions of both libtrace and libprotoident.

Started updating libwandbgp to be able to read bzip2 compressed files (such as the ones released by routeviews).

08

Jul

2011

The first half of the week was spent tidying up website and weekly report things and crossing things off my todo list which is getting quite short now which is nice, I also managed to not break Drupal this week which was nice.

Second half of the week was spent working with Jamie upgrading spectre and relic to Debian Squeeze. We also swapped the hardware of spectre and voodoo as voodoo was a little overspec'd which caused a bit of downtime for each. Upgrades were pretty successful and there were only a few minor issues which have been fixed. We also did an inventory of all the patch panels and cabling and Jamie went through and labeled all the switches in cacti and updated the cabling documentation.

06

Jul

2011

It's a busy day of software releases - libprotoident 2.0.2 has also been released!

This release further improves the range of protocols matched by libprotoident, as well as improving the rules for some existing ones. There is also a new tool included with libprotoident, lpi_live, that classifies flows as soon as possible (rather than waiting for the flow to expire, as lpi_protoident does) and thus is more useful for real-time analysis.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.2 here!

06

Jul

2011

Libtrace 3.0.11 has been released!

This release adds support for ECN bits in the TCP header, fixes a notorious bug where trace format auto-detection failed on small trace files and fixes several problems with BPF filters and the event API.

Note: due to the changes in the TCP header, some libtrace programs that examine the reserved bits in the TCP header may not build against libtrace 3.0.11 (especially any code that did so because we didn't support ECN previously!). We apologise for any inconvenience resulting from this change.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

04

Jul

2011

Wrote a simple AMP test to check for link activity before triggering a
chained test in order to help prevent bandwidth hogging tests from running
at the same time as other things on the machine. Seems to do the trick,
will wait for a configurable period for the link to be below a given level
of activity and eventually give up if that isn't reached.

Chased up a few more byte ordering issues with message IDs (related to
the previous fix, it had only done half the job) that were preventing
watchdogs messages from firing on little-endian clients connected to
little-endian servers.

Put together a small sample schedule testing a few popular websites with a
variety of tests and wrote up a quick script to report data usage to get
an idea what checking for updates, performing tests, reporting data etc
costs us. Will set the device up at home for a while and see what happens
outside the lab environment.

04

Jul

2011

Had a few problems processing Waikato I with all my analysis modules, mainly due to memory consumption. Did manage to get results for some of the analyses though and
worked on developing scripts to turn those results into useful graphs.

I blogged about some of the more interesting graphs here: http://www.wand.net.nz/content/messing-around-waikato-i

Found and fixed a libtrace bug where the automatic format detection would fail for small trace files - this one was bugging me for a while and cropped up again during the Waikato I analysis. Thinking about doing another release in the near future.