User login

Blogs

12

Jul

2011

Picked up working on SMTP state machines again this week. Decided the
first new step was to expand the flows I was learning from to include
those that didn't progress to the point of sending data (when testing
against a new dataset it was incorrectly dealing with flows truncated due
to greylisting etc). Unfortunately running it against that much extra data
meant it took longer than expected and had to be stopped for the upgrade
to spectre on Wednesday. While it was running I refactored a lot of the
code and spent time trying to make it both faster and more readable. After
the upgrade I also had to spend some time getting a recent version of R
running nicely again.

With the upgrade of relic I decided it was time to rewrite the emulation
network machine imaging and configuration control code to remove the
integration with ns2. It is now working again for imaging machines and can
configure basic networking for Linux hosts. Once I add in basic
configuration for FreeBSD/OpenBSD it will be at the same point as the old
system was. Hoping I can add in some more smarts and improve on the system
overall.

11

Jul

2011

Over the past week I added a couple more features to my network map such as an overview box and a debugging mode. The debugging mode just shows useful things like the frame rate, current scaling and translation offsets, and shows the outlines of nodes.
I think that I have now read and briefly summarised most of the relevant publications for my lit review.
Stayed at Hahei all weekend! Life of a student.

11

Jul

2011

Continued to have a few problems with processing large trace sets. Finally managed to get rid of all the memory leaks in my analysis code and fixed a tricky little libtrace bug that would cause processing to stop as soon as it hit a trace file less than 1 MB in size.

Started working on a web page to display all the graphs I'm creating nicely - http://www.wand.net.nz/~salcock/longitude/ . At the moment, only some of the Waikato I results are up, but will continue adding more results over the course of the week.

Tested and released new versions of both libtrace and libprotoident.

Started updating libwandbgp to be able to read bzip2 compressed files (such as the ones released by routeviews).

08

Jul

2011

The first half of the week was spent tidying up website and weekly report things and crossing things off my todo list which is getting quite short now which is nice, I also managed to not break Drupal this week which was nice.

Second half of the week was spent working with Jamie upgrading spectre and relic to Debian Squeeze. We also swapped the hardware of spectre and voodoo as voodoo was a little overspec'd which caused a bit of downtime for each. Upgrades were pretty successful and there were only a few minor issues which have been fixed. We also did an inventory of all the patch panels and cabling and Jamie went through and labeled all the switches in cacti and updated the cabling documentation.

06

Jul

2011

It's a busy day of software releases - libprotoident 2.0.2 has also been released!

This release further improves the range of protocols matched by libprotoident, as well as improving the rules for some existing ones. There is also a new tool included with libprotoident, lpi_live, that classifies flows as soon as possible (rather than waiting for the flow to expire, as lpi_protoident does) and thus is more useful for real-time analysis.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.2 here!

06

Jul

2011

Libtrace 3.0.11 has been released!

This release adds support for ECN bits in the TCP header, fixes a notorious bug where trace format auto-detection failed on small trace files and fixes several problems with BPF filters and the event API.

Note: due to the changes in the TCP header, some libtrace programs that examine the reserved bits in the TCP header may not build against libtrace 3.0.11 (especially any code that did so because we didn't support ECN previously!). We apologise for any inconvenience resulting from this change.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

04

Jul

2011

Wrote a simple AMP test to check for link activity before triggering a
chained test in order to help prevent bandwidth hogging tests from running
at the same time as other things on the machine. Seems to do the trick,
will wait for a configurable period for the link to be below a given level
of activity and eventually give up if that isn't reached.

Chased up a few more byte ordering issues with message IDs (related to
the previous fix, it had only done half the job) that were preventing
watchdogs messages from firing on little-endian clients connected to
little-endian servers.

Put together a small sample schedule testing a few popular websites with a
variety of tests and wrote up a quick script to report data usage to get
an idea what checking for updates, performing tests, reporting data etc
costs us. Will set the device up at home for a while and see what happens
outside the lab environment.

04

Jul

2011

Had a few problems processing Waikato I with all my analysis modules, mainly due to memory consumption. Did manage to get results for some of the analyses though and
worked on developing scripts to turn those results into useful graphs.

I blogged about some of the more interesting graphs here: http://www.wand.net.nz/content/messing-around-waikato-i

Found and fixed a libtrace bug where the automatic format detection would fail for small trace files - this one was bugging me for a while and cropped up again during the Waikato I analysis. Thinking about doing another release in the near future.

01

Jul

2011

As you may be aware, I'm working on a longitudinal study that examines the various trace sets in the WITS archive. I've now built most of the analysis software and have started running it on the traces, starting with Waikato I. Already a few interesting results are appearing which I felt might be worth sharing (apologies in advance for the .eps files).

Firstly, the [Bad link]. ICMP drops off massively in April 2005 - possibly due to an AMP box being shutdown?

I've also broken down ICMP traffic by type and code to create graphs for [Bad link] and [Bad link] ICMP packets.

[Bad link] for the quarter beginning March 1 show the start of a likely trend away from 576 byte packets.

[Bad link] is a very intriguing result. In 2005, the proportion of flows that used a rwin greater than 64 KB grew, but the proportion of flows using a window less than 10 KB also grew. Window scaling is not very widely employed at this time.

[Bad link] are no less interesting. 80% of flows used a window less than 64 KB, with over half in 2005 using 23 KB.

I've also looked at matching TCP SYNs to SYN ACKS in an attempt to identify TCP port scans. I've created a [Bad link] - we can see that the [Bad link] is actually dropping slowly over time. Counting the unique IP addresses that transmit unanswered SYNs also makes for [Bad link].

There are also some relatively conventional summaries of [Bad link] as well as [Bad link] and [Bad link] traffic. The plotted values are the result of taking the 90th percentile of the traffic rates measured over a 12 hour period.

Finally (for this post anyway), I've analysed the TTL used by both the inside (i.e. University) and outside (i.e. everyone else). Not only does this give us an indicator as to how many hops might be between the endpoints and our monitor, but it can also be used as a rudimentary OS fingerprinter. It turns out that the TTL distributions can change quite a lot over the course of one year!

For instance, check out [Bad link] in the March quarter. Looks like lots of packets with an original TTL of 255 have disappeared. However, looking at the [Bad link] shows that the number of flows where high TTLs were employed have barely changed.

On the University side, [Bad link] for the March quarter shows that most traffic exiting the University originated with a TTL of 64.

These graphs are just the starting point - there will be plenty more over the next few weeks - but hopefully that will give people a taste of what I'm up to at the moment. Any comments or suggestions (particularly in terms of how to visualise a lot of this data) are more than welcome.

01

Jul

2011

I'm back after taking a break from WAND (only in working capacity, pretty sure I lived up here for a couple of weeks studying...) for exams and to work on some other projects.

Spent the end of this week fixing Drupal bugs, pretty sure I've got all the major ones worked out (no more denied access for anonymous users!) this of course broke other things in Drupal but I think I've managed to sort most of those out now. Also made a few changes to the weekly report system which should make Shane's job easier.