User login

Blogs

13

Jun

2011

Continued trying to get a useful libprotoident comparison result using data captured live from the ISP capture point. Managed to solve some of my memory issues by reducing the amount allocated to the DAG card - can now run tests for a decent length of time without running into swap.

However, I was still finding that many of the DPI tools were performing surprisingly poorly even when working with simple HTTP flows. Managed to track this down to a rather obscure libtrace bug where the cached capture length is not reset when using a bpf filter in combination with event-based DAG capture.

While waiting for captures etc. to run, I continued working on the text for a paper on the topic. Mostly done in terms of background, methodology and evaluation techniques - just need to start putting some useful results in there.

07

Jun

2011

Started to really dig into OpenWRT this week. Downloaded and built the
build environment/toolchains to let me cross compile AMP for an OpenWRT
router and set about getting it going. The method of constructing packages
is quite similar to how Debian does it, so it was fairly easy to get the
Makefiles etc set up correctly. Ran into a few troubles with specifying
(and actually installing) dependencies and had to learn about "quilt" to
automatically patch the upstream source as part of the build process
(quilt seems like quite an interesting patch management tool).

An Ubiquiti AirRouter arrived on Friday for me to start testing with. I
moved my development over to using their SDK based around OpenWRT which
means we can keep the existing web interface though trades that off for an
older kernel version. Successfully flashed it with an image I had built
and managed to run the amplet client code! Getting config files into the
appropriate places doesn't quite work as expected so it exited instantly,
but it did actually work which is a confidence boosting start.

07

Jun

2011

Stoked to have finally submitted my doctoral thesis on Tuesday the 31st May. Very appreciative of the combined efforts of Tony, Bill and Sally Jo.

07

Jun

2011

Continued comparing libprotoident against various DPI-based solutions. Been trying to do some useful comparisons using ISP data (which has a much more interesting variety of traffic) but have been running into a few problems -- I can't capture full payload to disk, but running all of the traffic classifiers at the same time requires more memory than the capture box currently has.

If I disable the IP-based tracking that OpenDPI and PACE use, I can reduce the memory requirements enough to run a comparison test for a decent length of time. However, the classification accuracy of those tools drops massively, especially for P2P protocols, so the IP-based tracking is clearly more important than I had initially thought.

Set up and ran some performance tests for libprotoident and the DPI tools, measuring both CPU and memory usage.

Started writing up a draft paper on libprotoident -- not sure of a venue for it yet, but it will at least be a nice summary of all my comparative test results.

03

Jun

2011

Had the last week of my contract this week before I can sign a new one in 3 weeks. Spent this week pushing lots of fixes live, mostly centred around the weekly report system which should work a bit better now (it even sends out emails every week).

Got study week next week and then 2 weeks of exams so probably won't be around too much until exams are over.

03

Jun

2011

This week I wrote my interim report for 520. Also planned out what I need to get done next semester. I need to convert this into some sort of timeline at some stage. It was a busy week with presentations and tests to prepare for. One last assignment left now at this stage but that will be next weeks job.

Weekend off!

31

May

2011

Extracted some more data from ISP traces to use in predicting client MTA
and possible spam status. It isn't obvious that throwing more data at it
than I already had has helped, but even with the limited number of flows
with client MTAs that I can identify it is accurately predicting the MTA
70-80% of the time (on flows where it could be one of multiple MTAs -
ignoring flows that traverse a link used only by a single MTA).

Also ran some ISP traces with a full traffic mix against my machine
generated from a subset of SMTP flows. It missed lots of prematurely
terminated SMTP flows (eg ones that ended immediately after the HELO/EHLO)
because the training data only included connections that sent DATA. Of the
49,000 non SMTP flows it classified 158 as matching SMTP, these were
entirely FTP and POP3 flows which are quite similar.

Started reading up on OpenWRT in preparation for getting AMP to run on it.
Should hopefully have a device to test on in the next week or so but in
the meantime I need to get my head around how it all fits together.

30

May

2011

Ran some experiments to compare the accuracy of libprotoident with the DPI-based traffic classification tools I've managed to get my hands on. Much of this time was spent figuring out various quirks with the other tools that was causing them to perform more poorly than expected -- e.g, TIE fails to identify HTTP by default if the GET request is more than a couple of hundred bytes.

Finally managed to get everything working properly towards the end of the week and had completed a preliminary study using some full-payload Auckland traces we'd taken last year. Results were very promising: using the PACE classifications as our ground truth, only 0.9% of traffic is not correctly identified by libprotoident, compared with 1.5% for OpenDPI and 12.4% for the L7 filter module included with TIE.

28

May

2011

Had a play around with the google maps api. I need to visualise exports from a given country (starting with NZ) to all other countries for a group project. I turned a list of countries in to longitudes and latitudes using the google geolocation api. Was rather easy - just loop through all of the countries and make an api request in a rate limited way. Then, using export data from stats New Zealand, I drew weighted lines from NZ to all other countries. The resulting shape kind of looks like a butterfly.

View image

This is only the first iteration. The next step is to draw paths through the ocean that split off into countries. Similar idea to Minard's famous French wine exports drawing:

View Image

I also made a start on my 520 interim report, polished off two assignments and made a couple improvements to my network map.

24

May

2011

Managed to have a busy week this week with 5 unrelated software problems causing outages on different servers throughout the week that needed attention to fix. Erg was the first to die on Saturday which I brought back Monday morning followed by wagyu having a short outage on Tuesday. Warlock got into an unhappy state twice requiring a reboot to bring it back from the dead. Also the VPS hosting the KAREN wiki that I look after for REANNZ decided to go down as well on Thursday which came back after a fsck and a restart.

Looking forward to a "quiter" week next work catching up on uni assignments.