User login

Search Projects

Project Members

Shane Alcock admin

Libprotoident

Libprotoident is a library that performs application layer protocol identification for flows. Unlike many techniques that require capturing the entire packet payload, only the first four bytes of payload sent in each direction, the size of the first payload-bearing packet in each direction and the TCP or UDP port numbers for the flow are used by libprotoident. Libprotoident features a very simple API that is easy to use, enabling developers to quickly write code that can make use of the protocol identification rules present in the library without needing to know anything about the applications they are trying to identify.

31

Jan

2012

At NZNOG 2012, I presented some slides showing a decrease of P2P traffic following the Copyright Amendment Act coming into effect in New Zealand in September 2011. By contrast, the same analysis showed a significant increase in Tunneling, FTP and Remote Access traffic. These results generated a lot of interest, so I am using this blog post to discuss our methodology and results in more detail.

31

Jan

2012

Last Friday, I presented a talk at NZNOG 2012 about libprotoident and presented some results showing the impact of the Copyright Amendment Act on New Zealand residential DSL usage.

The slide set from this talk has been attached to this blog post for anyone who missed the talk or wishes to look over the results in more detail.

Feel free to get in touch with me if you have any questions or comments about libprotoident or the results presented in the talk.

31

Jan

2012

Made a few tweaks and changes to my NZNOG slides based on feedback from last week's practice run. Also spent some time working with some of the latest ISP capture and adding new rules to libprotoident based on that.

Spent Wed-Fri at NZNOG. The main program was very high quality this year and it proved to be a rather educational experience. In particular, my interest was piqued by OpenFlow and how it could be used in combination with libprotoident to implement interesting routing policies. My own talk went reasonably well and seemed to catch people's attention. We'll wait and see whether that translates into anything more tangible over the next wee while.

16

Jan

2012

Finally found and fixed the bug that was causing the occasional trace file to be truncated when written to disk. Having done that, I released libtrace 3.0.13 on Monday.

Worked with Nevil to get a test capture up and running on his capture box in Auckland. After a couple of false starts, we managed to successfully capture a day's worth of trace without issues.

Set up a Fedora machine for testing libtrace prior to subsequent releases, as it has become apparent that testing on just Debian and Ubuntu is insufficient. Will hopefully replace with a virtual machine once the new emulation network is up and running.

Started working on a possible presentation for NZNOG, mostly about libprotoident again.

Spent a little bit of time reading over my extended NAT sessions paper, making a few edits here and there.

14

Dec

2011

Libprotoident 2.0.4 has been released today.

This release adds support for 9 new protocols (including QQLive, Paltalk and DriveShare). It also improves the rules for many existing protocols and adds a couple of new features to the lpi_live tool.

The full list of changes can be found in the libprotoident ChangeLog.

Download libprotoident 2.0.4 here!

06

Dec

2011

Received a new version of NAVL from Vineyard, but unfortunately there is still a problem with double entries in the internal flow cache. I've created a NAVL-only version of the program I've been using and sent that off to them along with a small sample trace that should replicate the problem.

Got some good news in that our ATNAC paper has been recommended for publication in Telecommunication Systems. However, we need at least 40% new content on top of what we've already got and it needs to be ready by Jan 22. Richard suggested we chuck in the work I did measuring outbound TCP and UDP sessions for the SPNAT study, so I started running the analysis against some more recent traces and changing the introductory material to talk about outbound sessions as well as inbound.

Got my degradation graphs looking the way I wanted them to, but a bit of extra analysis revealed that I may have set my sleeper threshold too low. Most of the "sleeping" periods were only just longer than the original threshold of 5 minutes. I've repeated some of the earlier analysis with a threshold of 30 minutes to see how much of a difference that makes.

28

Nov

2011

Continued looking into properties of sleeper traffic, primarily the rate at which sleeper traffic quantities degrade as the host continues to be idle. This has proved a bit tricky to visualise well, but finally managed to come up with what I think should be a useful graphing approach. This did require a lot of battling with R, though.

The fixed version of NAVL was not available last week, but I was able to continue looking at cases where PACE was able to identify traffic that libprotoident could not. Brad set me up with a Windows VM so that I could download various apps and capture traffic while using them, so that I can confirm PACE's classifications and add or update libprotoident's rules so that we can match the traffic as well. This meant I got to have a bit of fun playing Second Life and hanging out in chatrooms....

Started moving towards a new release of libprotoident, seeing as I've now added or updated the rules for quite a few protocols.

21

Nov

2011

Managed to create a new model for use with the Bernaille traffic classification technique, based on an hour of ISP traffic and using PACE to determine ground truth. The model does not perform much better than the default one I tested last week, despite including a few extra protocols.

Developed a new technique for comparing the various traffic classification schemes. My main problem is that even the commercial tools are not reliable enough to act as a genuine ground truth, so it becomes difficult to evaluate the accuracy of any given approach. My new approach evaluates each tool by comparing the classifications against the results produced by each other tool in turn, treating all flows that are unknown or classified differently as failure cases. The average failure rate is then calculated across all the tools compared against to produce an estimated accuracy rating for the evaluated tool.

So far, the results produced by this comparison approach have matched my expectations (libprotoident and PACE have lower failure rates, nmap has the highest) and have also highlighted the high quality of libprotoident's classifications. Hopefully, we will continue to have good results when NAVL is added to the mix.

On that note, still waiting on Vineyard to provide me with a binary that fixes the bug I reported last week - it has been acknowledged as a bug and are in the process of testing the fix now.

14

Nov

2011

Started collating together the results of my analysis of dark and sleeper traffic in the ISP traces. It's not finished yet, but the results I have so far can be viewed at http://www.wand.net.nz/~salcock/sleepers/

CCR rejected my libprotoident paper, primarily due to a reviewer stating that we had not compared against the "state of the art" described in a paper from 2006 (http://www-rp.lip6.fr/site_npa/site_rp/_publications/737-conextFinal.pdf). This particular technique requires no packet payload, but is only able to identify 10 different TCP application protocols (although I can supposedly create new models for other TCP applications).

I tested the default models against some ISP traffic and found that it performed much better than I had expected, but was still less accurate than the weakest of the OSS DPI techniques. Their failure rate (in terms of misclassified bytes) was 24%, compared with 4.5% for libprotoident.

Started integrating Vineyard's NAVL library into my traffic classification evaluation tool. Started out OK, but ran into a few problems with not being able to force NAVL to expire internal entries for UDP flows when I have decided the flow has ended. This creates a problem if the 5-tuple reappears later, as NAVL returns an error when I try to create a new NAVL connection for that flow because NAVL believes the flow already exists. I've filed a support request, so hopefully I'll get some sort of solution in the next day or two.

Continued integrating Simon's OSPF code into libtrace.

07

Nov

2011

Started looking into the traffic sent to "sleeper" hosts, i.e. IP addresses that have been active but are now inactive. Still putting together the initial results, but there is definitely a difference between the traffic observed heading to "dark" hosts vs the traffic observed heading to sleepers.

During the sleeper analysis, I've been able to improve a few of the libprotoident rules to correctly match more of the traffic I've been looking at.

Began integrating Simon's OSPF parsing code into libtrace. Been slightly trickier than I had anticipated due to major differences between OSPFv2 (which Simon's code parses) and OSPFv3 (which we may want to parse in future).

Had a brief phone meeting with Vineyard Networks. They've agreed to give us access to their NAVL library for evaluation.

04

Oct

2011

Libprotoident 2.0.3 has been released today.

This release adds support for 13 new protocols (including RADIUS, Akamai and Youku) and 3 new categories (Logging, Printing and Translation). It also improves the rules for some existing protocols and fixes a few bugs.

The included tools have all been updated to support analysis of IPv6 traffic and also provide more options for determining the direction of analysed packets.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.3 here!

19

May

2011

Libprotoident 2.0.1 has been released!

This release fixes a number of bugs in 2.0.0, as well as adding support for new application protocols and improving the rules for many existing ones.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.1 here!

18

May

2011

I have created Trac sites for both the libprotoident and BSOD projects, so it is now possible to file tickets to report bugs or request features for either of these projects through the Trac system, rather than having to contact me directly.

The Trac sites also feature wikis which I intend to use to provide more extensive documentation for these projects, e.g. explanations of the protocols supported by libprotoident. At the moment, this is a work in progress but hopefully will get fleshed out over time.

The BSOD trac: http://wand.net.nz/trac/bsod/
The libprotoident trac: http://wand.net.nz/trac/libprotoident