User login

Shane Alcock's Blog

25

Jul

2011

Released maji-1.0.2 on Monday.

Continued processing trace sets to generate results for my graphs. I soon realised that manually updating the scripts that convert the results into graphs was taking a horrendously long time (and was also incredibly tedious), so started working on a different script that could automatically generate the bash and gnuplot scripts I had been using. I can then just add a single line to the script for each new dataset and the graphs will be created from there.

This hasn't been entirely straightforward - each different type of graph has subtle differences that prevent me from using a single generic "make CDF" or "make time series" function. Still, it seems to be working well, just need to add code for the last few analysis types that I've done.

Found out over the weekend that the libtrace paper was rejected by IMC again, but the inbound sessions paper was accepted by ATNAC.

18

Jul

2011

After a two year development hiatus, a new version of maji (our libtrace-based IPFIX meter) has finally been released.

This release adds support for encrypting IPFIX records exported using the TCP and UDP transports, fixes some bugs observed when measuring IPv6 traffic, adds new information elements for ICMPv6 and also fixes a few errors and warnings that have come about due to changes in supporting software over the past couple of years.

We would like to offer special thanks to Benjamin Black and Rong Zheng, who both contributed code towards this release of maji.

The full list of changes is described in the maji ChangeLog.

Download maji-1.0.2 here!

18

Jul

2011

Continued processing Waikato traces and adding more graphs to http://www.wand.net.nz/~salcock/longitude/ . Had a few segfault-related hiccups along the way which has slowed things down quite a bit.

Worked on a new analysis test that would divide traffic by source and destination AS using MIBs downloaded from routeviews. Turned out that libwandbgp couldn't really support maintaining a BGP route table from routeviews data for any length of time, so had to re-write a decent chunk of the library to be able to do so. Still not quite working yet, but starting to get there.

Received some patches for both libtrace and maji from one of our users, which I checked and integrated back into our software for future release.

11

Jul

2011

Continued to have a few problems with processing large trace sets. Finally managed to get rid of all the memory leaks in my analysis code and fixed a tricky little libtrace bug that would cause processing to stop as soon as it hit a trace file less than 1 MB in size.

Started working on a web page to display all the graphs I'm creating nicely - http://www.wand.net.nz/~salcock/longitude/ . At the moment, only some of the Waikato I results are up, but will continue adding more results over the course of the week.

Tested and released new versions of both libtrace and libprotoident.

Started updating libwandbgp to be able to read bzip2 compressed files (such as the ones released by routeviews).

06

Jul

2011

It's a busy day of software releases - libprotoident 2.0.2 has also been released!

This release further improves the range of protocols matched by libprotoident, as well as improving the rules for some existing ones. There is also a new tool included with libprotoident, lpi_live, that classifies flows as soon as possible (rather than waiting for the flow to expire, as lpi_protoident does) and thus is more useful for real-time analysis.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.2 here!

06

Jul

2011

Libtrace 3.0.11 has been released!

This release adds support for ECN bits in the TCP header, fixes a notorious bug where trace format auto-detection failed on small trace files and fixes several problems with BPF filters and the event API.

Note: due to the changes in the TCP header, some libtrace programs that examine the reserved bits in the TCP header may not build against libtrace 3.0.11 (especially any code that did so because we didn't support ECN previously!). We apologise for any inconvenience resulting from this change.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

04

Jul

2011

Had a few problems processing Waikato I with all my analysis modules, mainly due to memory consumption. Did manage to get results for some of the analyses though and
worked on developing scripts to turn those results into useful graphs.

I blogged about some of the more interesting graphs here: http://www.wand.net.nz/content/messing-around-waikato-i

Found and fixed a libtrace bug where the automatic format detection would fail for small trace files - this one was bugging me for a while and cropped up again during the Waikato I analysis. Thinking about doing another release in the near future.

01

Jul

2011

As you may be aware, I'm working on a longitudinal study that examines the various trace sets in the WITS archive. I've now built most of the analysis software and have started running it on the traces, starting with Waikato I. Already a few interesting results are appearing which I felt might be worth sharing (apologies in advance for the .eps files).

Firstly, the [Bad link]. ICMP drops off massively in April 2005 - possibly due to an AMP box being shutdown?

I've also broken down ICMP traffic by type and code to create graphs for [Bad link] and [Bad link] ICMP packets.

[Bad link] for the quarter beginning March 1 show the start of a likely trend away from 576 byte packets.

[Bad link] is a very intriguing result. In 2005, the proportion of flows that used a rwin greater than 64 KB grew, but the proportion of flows using a window less than 10 KB also grew. Window scaling is not very widely employed at this time.

[Bad link] are no less interesting. 80% of flows used a window less than 64 KB, with over half in 2005 using 23 KB.

I've also looked at matching TCP SYNs to SYN ACKS in an attempt to identify TCP port scans. I've created a [Bad link] - we can see that the [Bad link] is actually dropping slowly over time. Counting the unique IP addresses that transmit unanswered SYNs also makes for [Bad link].

There are also some relatively conventional summaries of [Bad link] as well as [Bad link] and [Bad link] traffic. The plotted values are the result of taking the 90th percentile of the traffic rates measured over a 12 hour period.

Finally (for this post anyway), I've analysed the TTL used by both the inside (i.e. University) and outside (i.e. everyone else). Not only does this give us an indicator as to how many hops might be between the endpoints and our monitor, but it can also be used as a rudimentary OS fingerprinter. It turns out that the TTL distributions can change quite a lot over the course of one year!

For instance, check out [Bad link] in the March quarter. Looks like lots of packets with an original TTL of 255 have disappeared. However, looking at the [Bad link] shows that the number of flows where high TTLs were employed have barely changed.

On the University side, [Bad link] for the March quarter shows that most traffic exiting the University originated with a TTL of 64.

These graphs are just the starting point - there will be plenty more over the next few weeks - but hopefully that will give people a taste of what I'm up to at the moment. Any comments or suggestions (particularly in terms of how to visualise a lot of this data) are more than welcome.

27

Jun

2011

Spent the entire week implementing various analyses to run over our various trace sets. Aside from a couple, everything on my list is now implemented and it is just a matter of getting them to run over all the traces and turn the output into interesting graphs.

Converted my old object extraction tool into a library with a usable external API and reimplemented the tool using that API. The library was then used to implement an HTTP and SMTP object analysis for the above study.

Made Waikato V available on the WITS FTP site - we've had a couple of requests for more recent traces and there was enough space on mojo to fit Waikato V.

20

Jun

2011

Finished up my report on comparing libprotoident to other traffic classifiers. Anyone interested in reading it can find it here: http://www.wand.net.nz/~salcock/drafts/lpi_report.pdf

Overall, we do pretty well - we easily outperform the OSS DPI tools in just about every category and are not really that far off the commercial PACE engine. Remember, we're also only working with 4 bytes of payload too, whereas they have the whole packet!

Started working on a system for processing all our traces and extracting various stats about the traffic, flows, hosts etc. By the time you read this, you should have seen the email I sent to the WAND list describing what I'm looking at so far. I've implemented most of the things on my list so far, but the amount of output generated could be a bit of a problem. Started working on making my output a bit more efficient, i.e. instead of reporting the duration of every flow, doing some binning and reporting the number of flows that fall into each bin.