User login

The Case Against L7 Filter

07

Dec

2012

L7 Filter is used as a source of ground truth in the traffic classification field because it has been around for a long time and is widely known. However, my experiences with L7 Filter had raised a few questions in my mind with regard to its accuracy. After looking online, I did not find any evidence that L7 Filter is actually an accurate or reliable traffic classifier. In this blog post, I present some preliminary results from my own investigation into the correctness (or lack thereof) of L7 Filter's classifications using packet traces containing traffic for only a single known application.

L7 Filter is an open-source application-level traffic classifier that uses deep packet inspection to identify the application protocol for a given network flow. L7 Filter was originally designed to be used with Linux NetFilter in kernel-space as a traffic shaping and bandwidth limiting tool, although much of the current development effort is focused on a user-space version. L7 Filter compares the packet payload with application signatures expressed as regular expressions to determine the application being used by a traffic flow.

Public releases of L7 Filter date back to 2003, so it predates most other open-source traffic classifiers by several years. Because of its longevity and a resulting widespread awareness of the software, L7 Filter has become something of a de-facto source of ground truth when evaluating, validating or even simply developing new traffic classification techniques and tools. It is very easy to find papers in this field that either use L7 Filter for validation (1, 2) or as a source of protocol-matching patterns to derive their own classification rules (3, 4).

However, my own experiences with L7 Filter when attempting to validate libprotoident did not instill a lot of confidence, especially when attempting to classify contemporary Internet traffic. These doubts were further compounded when I noticed that the last public release of the patterns used to match application protocols was in 2009! The research community continues to treat L7 Filter as a suitable source of ground truth (most likely because it is what has always been used for ground truth), despite the fact that there is no published data showing that L7 Filter itself is accurate and reliable.

As a result, I have investigated the classification accuracy of L7 Filter in a series of scenarios where the ground truth is already known. To do this, I used a variety of common Internet applications and captured packet traces of the resulting traffic. The applications that I used for this investigation were:
* Basic web browsing via HTTP (no video)
* Browsing News Websites, including watching news video clips
* YouTube (both watching a video and uploading one)
* Facebook (using a secure login and HTTPS)
* World of Warcraft
* uTorrent (including using uTP)
* iTunes Store (listening to podcasts)
* Email (IMAPS, POP3S and SMTP protocols)
* Google Talk (using XMPPS)
* Jabber (using XMPP)
* DNS (capturing just DNS traffic while web browsing)
* Steam (downloading a game)
* Team Fortress 2
* Second Life
* Spotify
* FTP (downloading a file)
* RDP
* Skype
* SSH

I captured a separate trace file for each application and made every effort to ensure that I was only capturing traffic for that application (using BPF filters). Each trace was analysed using L7 Filter, libprotoident, nDPI and PACE. The L7 Filter analysis was run using the Traffic Identification Engine which helpfully included an L7 Filter module. The version of PACE was an older release from 2011 that was provided to us by ipoque to aid us in validating libprotoident. Once the trace analysis was complete, the classifications produced by each tool were merged into a single output file and the results were manually inspected to determine whether the classifications were correct.

I used a simple ternary system to determine whether a traffic classification tool was successful at identifying a particular application. If the tool correctly identified all flows directly associated with the application (i.e. not including any required DNS lookups) then the classification was deemed a "success". If more than 5% of the individual flows or more than 5% of the observed were incorrectly classified by the tool, then the tool had "failed" at identifying that application. The third case covered instances where the tool did not achieve 100% success but only failed on a small proportion of traffic - this was to highlight situations where the tool was generally correct but had the occasional blind-spot, e.g. the pattern used to match that application had minor errors or did not account for all valid use cases.

The results are summarised in the attached PDF file. L7 Filter was surprisingly poor at identifying most common applications used on the Internet today, even including basic web browsing over HTTP. For example, a number of obvious HTTP flows were classified as Unknown by L7 Filter. L7 Filter also frequently misclassified SSL traffic (i.e. HTTPS, POP3S, IMAPS, XMPPS) as Skype and did not understand streaming video protocols, e.g. RTMP.

The main conclusion to draw here is that L7 Filter is NOT the best option available for determining ground truth classifications for modern Internet traffic. I would be wary of any research into traffic classification that relies on L7 Filter for either validation or protocol matching rules. Similarly, ground truth projects that are heavily reliant on the patterns provided by L7 Filter may wish to re-evaluate the techniques that they are using to determine ground truth and consider doing more cross-validation with some of the other classifiers out there. For example, our results show that nDPI and libprotoident both perform much better than L7 Filter and are freely available for anyone to use.

The next step is to write these results up into a format suitable for peer-reviewed publication. The paper will contain a lot more detail about the validation process and the applications that we used to evaluate L7 Filter. In the meantime, feel free to direct any questions or comments to us via contact [at] wand [dot] net [dot] nz.

AttachmentSize
justtable.pdf17.29 KB
Groups: