User login

Blogs

14

Jun

2011

Spent most of the week getting better acquainted with the build process
for both packages and for firmware images. I got an AMP package building
fine within the environment last week but getting it running properly also
needs the supporting configuration files to be installed in the right
place with the correct file permissions. The filesystem layout is a bit
different to normal and (in most cases) is read only so I have to make my
changes at build time. Took me a while to discover that the final script
run before building the image clobbers all my permissions changes - had to
put in a few exceptions, which meant making changes outside of my
packages, which I would prefer not to have to do.

Also spent some time dealing with getting the ntpclient working properly
in my image. The slightly newer version I'm using accepts different
arguments to the version the Ubiquiti config generation binary blob
expects. Also noticed that this clobbers or adds to various configuration
files that are provided by the base packages. Chasing this also made
explicit to me the separation between firmware and configuration as stored
on the device itself, and the different ways that each may be updated.

13

Jun

2011

I spent the week finishing of my exports visualisation assignment and presenting it to the class.

You can take a look here:
http://joeloughton.com/blog/testing/exports/?s=view1

13

Jun

2011

Continued trying to get a useful libprotoident comparison result using data captured live from the ISP capture point. Managed to solve some of my memory issues by reducing the amount allocated to the DAG card - can now run tests for a decent length of time without running into swap.

However, I was still finding that many of the DPI tools were performing surprisingly poorly even when working with simple HTTP flows. Managed to track this down to a rather obscure libtrace bug where the cached capture length is not reset when using a bpf filter in combination with event-based DAG capture.

While waiting for captures etc. to run, I continued working on the text for a paper on the topic. Mostly done in terms of background, methodology and evaluation techniques - just need to start putting some useful results in there.

07

Jun

2011

Started to really dig into OpenWRT this week. Downloaded and built the
build environment/toolchains to let me cross compile AMP for an OpenWRT
router and set about getting it going. The method of constructing packages
is quite similar to how Debian does it, so it was fairly easy to get the
Makefiles etc set up correctly. Ran into a few troubles with specifying
(and actually installing) dependencies and had to learn about "quilt" to
automatically patch the upstream source as part of the build process
(quilt seems like quite an interesting patch management tool).

An Ubiquiti AirRouter arrived on Friday for me to start testing with. I
moved my development over to using their SDK based around OpenWRT which
means we can keep the existing web interface though trades that off for an
older kernel version. Successfully flashed it with an image I had built
and managed to run the amplet client code! Getting config files into the
appropriate places doesn't quite work as expected so it exited instantly,
but it did actually work which is a confidence boosting start.

07

Jun

2011

Stoked to have finally submitted my doctoral thesis on Tuesday the 31st May. Very appreciative of the combined efforts of Tony, Bill and Sally Jo.

07

Jun

2011

Continued comparing libprotoident against various DPI-based solutions. Been trying to do some useful comparisons using ISP data (which has a much more interesting variety of traffic) but have been running into a few problems -- I can't capture full payload to disk, but running all of the traffic classifiers at the same time requires more memory than the capture box currently has.

If I disable the IP-based tracking that OpenDPI and PACE use, I can reduce the memory requirements enough to run a comparison test for a decent length of time. However, the classification accuracy of those tools drops massively, especially for P2P protocols, so the IP-based tracking is clearly more important than I had initially thought.

Set up and ran some performance tests for libprotoident and the DPI tools, measuring both CPU and memory usage.

Started writing up a draft paper on libprotoident -- not sure of a venue for it yet, but it will at least be a nice summary of all my comparative test results.

03

Jun

2011

Had the last week of my contract this week before I can sign a new one in 3 weeks. Spent this week pushing lots of fixes live, mostly centred around the weekly report system which should work a bit better now (it even sends out emails every week).

Got study week next week and then 2 weeks of exams so probably won't be around too much until exams are over.

03

Jun

2011

This week I wrote my interim report for 520. Also planned out what I need to get done next semester. I need to convert this into some sort of timeline at some stage. It was a busy week with presentations and tests to prepare for. One last assignment left now at this stage but that will be next weeks job.

Weekend off!

31

May

2011

Extracted some more data from ISP traces to use in predicting client MTA
and possible spam status. It isn't obvious that throwing more data at it
than I already had has helped, but even with the limited number of flows
with client MTAs that I can identify it is accurately predicting the MTA
70-80% of the time (on flows where it could be one of multiple MTAs -
ignoring flows that traverse a link used only by a single MTA).

Also ran some ISP traces with a full traffic mix against my machine
generated from a subset of SMTP flows. It missed lots of prematurely
terminated SMTP flows (eg ones that ended immediately after the HELO/EHLO)
because the training data only included connections that sent DATA. Of the
49,000 non SMTP flows it classified 158 as matching SMTP, these were
entirely FTP and POP3 flows which are quite similar.

Started reading up on OpenWRT in preparation for getting AMP to run on it.
Should hopefully have a device to test on in the next week or so but in
the meantime I need to get my head around how it all fits together.

30

May

2011

Ran some experiments to compare the accuracy of libprotoident with the DPI-based traffic classification tools I've managed to get my hands on. Much of this time was spent figuring out various quirks with the other tools that was causing them to perform more poorly than expected -- e.g, TIE fails to identify HTTP by default if the GET request is more than a couple of hundred bytes.

Finally managed to get everything working properly towards the end of the week and had completed a preliminary study using some full-payload Auckland traces we'd taken last year. Results were very promising: using the PACE classifications as our ground truth, only 0.9% of traffic is not correctly identified by libprotoident, compared with 1.5% for OpenDPI and 12.4% for the L7 filter module included with TIE.