User login

Blogs

01

Jul

2011

As you may be aware, I'm working on a longitudinal study that examines the various trace sets in the WITS archive. I've now built most of the analysis software and have started running it on the traces, starting with Waikato I. Already a few interesting results are appearing which I felt might be worth sharing (apologies in advance for the .eps files).

Firstly, the [Bad link]. ICMP drops off massively in April 2005 - possibly due to an AMP box being shutdown?

I've also broken down ICMP traffic by type and code to create graphs for [Bad link] and [Bad link] ICMP packets.

[Bad link] for the quarter beginning March 1 show the start of a likely trend away from 576 byte packets.

[Bad link] is a very intriguing result. In 2005, the proportion of flows that used a rwin greater than 64 KB grew, but the proportion of flows using a window less than 10 KB also grew. Window scaling is not very widely employed at this time.

[Bad link] are no less interesting. 80% of flows used a window less than 64 KB, with over half in 2005 using 23 KB.

I've also looked at matching TCP SYNs to SYN ACKS in an attempt to identify TCP port scans. I've created a [Bad link] - we can see that the [Bad link] is actually dropping slowly over time. Counting the unique IP addresses that transmit unanswered SYNs also makes for [Bad link].

There are also some relatively conventional summaries of [Bad link] as well as [Bad link] and [Bad link] traffic. The plotted values are the result of taking the 90th percentile of the traffic rates measured over a 12 hour period.

Finally (for this post anyway), I've analysed the TTL used by both the inside (i.e. University) and outside (i.e. everyone else). Not only does this give us an indicator as to how many hops might be between the endpoints and our monitor, but it can also be used as a rudimentary OS fingerprinter. It turns out that the TTL distributions can change quite a lot over the course of one year!

For instance, check out [Bad link] in the March quarter. Looks like lots of packets with an original TTL of 255 have disappeared. However, looking at the [Bad link] shows that the number of flows where high TTLs were employed have barely changed.

On the University side, [Bad link] for the March quarter shows that most traffic exiting the University originated with a TTL of 64.

These graphs are just the starting point - there will be plenty more over the next few weeks - but hopefully that will give people a taste of what I'm up to at the moment. Any comments or suggestions (particularly in terms of how to visualise a lot of this data) are more than welcome.

01

Jul

2011

I'm back after taking a break from WAND (only in working capacity, pretty sure I lived up here for a couple of weeks studying...) for exams and to work on some other projects.

Spent the end of this week fixing Drupal bugs, pretty sure I've got all the major ones worked out (no more denied access for anonymous users!) this of course broke other things in Drupal but I think I've managed to sort most of those out now. Also made a few changes to the weekly report system which should make Shane's job easier.

29

Jun

2011

Got my AMP package for Ubiquiti AirOS to the point where it will check
for updates on startup before running (with a small random offset), checks
for updates to AMP configs and checks for updates to the firmware image at
random but known times of day. Should there be an update it will apply it
and restart anything that needs to be.

Looking at having certain tests wait for the link to be idle before
running themselves seems to be best accomplished by using chained tests -
the first one can check the traffic on the link and delay as required,
before either aborting or allowing the chain to continue. Started to write
up a simple version of this test and it looks like it should do the trick.

28

Jun

2011

Presented my talk to the Waikato branch of the NZCS on Thursday. Went really well and people were really interested. In fact, they were so interested that they let me talk for an hour and a half! A big thanks to Brad who came along and was a very helpful technical assistant!

27

Jun

2011

I collected some more useful publications for my lit review and started looking at ways to produce an overview of the network map.

27

Jun

2011

Spent the entire week implementing various analyses to run over our various trace sets. Aside from a couple, everything on my list is now implemented and it is just a matter of getting them to run over all the traces and turn the output into interesting graphs.

Converted my old object extraction tool into a library with a usable external API and reimplemented the tool using that API. The library was then used to implement an HTTP and SMTP object analysis for the above study.

Made Waikato V available on the WITS FTP site - we've had a couple of requests for more recent traces and there was enough space on mojo to fit Waikato V.

21

Jun

2011

Got AMP running happily on the Ubiquiti AirRouter and reporting results to
another machine. There were a few byte ordering issues with the AirRouter
being MIPS and the collector running on an x86 machine, but most of the
work here had already been planned for so I wasn't required to make
wholesale changes to get it running. Still had to spend a bit of time
tracing through the code checking what values were being used where and
making sure all communications were appropriately byte swapped.

Some changes in between libcurl versions were throwing off results
generated by the http2 test which had to be tracked down and fixed to get
it running on the device.

Investigated in greater depth the init system used for the AirRouter and
how to get AMP running on startup. Looks to be a few options on how this
can be done, but I think I've figured out the nicest approach to get it
doing what I want.

20

Jun

2011

Finished up my report on comparing libprotoident to other traffic classifiers. Anyone interested in reading it can find it here: http://www.wand.net.nz/~salcock/drafts/lpi_report.pdf

Overall, we do pretty well - we easily outperform the OSS DPI tools in just about every category and are not really that far off the commercial PACE engine. Remember, we're also only working with 4 bytes of payload too, whereas they have the whole packet!

Started working on a system for processing all our traces and extracting various stats about the traffic, flows, hosts etc. By the time you read this, you should have seen the email I sent to the WAND list describing what I'm looking at so far. I've implemented most of the things on my list so far, but the amount of output generated could be a bit of a problem. Started working on making my output a bit more efficient, i.e. instead of reporting the duration of every flow, doing some binning and reporting the number of flows that fall into each bin.

20

Jun

2011

I had a look at ways of visualising bandwidth in an effective way on a network map. I am experimenting more with an idea that incorporates width, height and colour of edge sections. Where width codes the bandwidth utilisation percentage, height codes the actual throughput and the colour codes some categorical severity measure.

See: http://joeloughton.com/blog/wp-content/uploads/2011/06/bandwidth2.png

Coding the height of a line in a map that has zooming capabilities does not work well. I have tried scaling the line height in proportion to the current zoom level which seems to be working well.

20

Jun

2011

My practice presentation for NZCS at the Friday night WAND meeting attracted a full room of people and received rave reviews on Twitter, with 20+ tweets (sorry no screenshots). I also received some great feedback and was stoked to see that Angela and a few other non-WANDees accepted Tony's invitation of attending the talk.