User login

Search Projects

Project Members

Shane Alcock admin

Libtrace

Libtrace is a library for both capturing and processing packet traces. It supports a variety of common trace formats, including pcap, ERF, live DAG capture, native Linux and BSD sockets, TSH and legacy ERF formats. Libtrace also supports reading and writing using several different compression formats, including gzip, bzip2 and lzo. Libtrace uses a multi-threaded approach for decompressing and compressing trace files to improve trace processing performance on multi-core CPUs.

The libtrace API provides functions for accessing the headers in a packet directly,
up to and including the transport header.

Libtrace can also output packets using any supported output trace format, including
pcap, ERF, DAG transmit and native sockets.

Libtrace is bundled with several tools for performing common trace processing and analysis tasks. These include tracesplit, tracemerge, traceanon, tracepktdump and tracereport (amongst others).

10

Dec

2012

Libtrace:
Managed to get native BPF socket capture exporting correctly over the RT protocol. Changed the build system to make it possible to export captures taken using a native socket interface over RT to a machine running a different OS to the capture host, e.g. capture using Linux Native, export to a FreeBSD box.

WDCap:
WDCap now builds and runs on both Mac OS X and FreeBSD. Also changed the way the disk output module names files, based on some code submitted by Alistair King. You now specify your output filename format using strftime-style conversion modifiers, which offers a bit more flexibility to users rather than them being stuck with our particular file naming convention.

lpi_collector:
Continued working closely with Meenakshee on the new collector. Designed a binary format for exporting our collector messages called the libprotoident collector protocol (or LPICP for short).

L7 Filter:
Finished collecting traces for most of the protocols I wanted to test with L7 Filter and collated the initial results. Wrote a blog post about it (https://secure.wand.net.nz/content/case-against-l7-filter) and started working on a paper.

03

Dec

2012

Back into the swing of things this week. Continued collecting traces of various popular Internet applications to use for validating L7 Filter. So far, L7 Filter is very disappointing - it cannot even correctly classify some basic HTTP flows and often misclassifies SSL traffic as Skype.

Worked with Meenakshee to develop a proper LPI collector that we can run on passive monitors and write live application stats to a database (ideally using Nathan's code). The new collector will use libwandevent and export its results over the network rather than via stdout. To help with this, I extracted the counter / statistic management code from the old lpi_live tool and tidied it up for more general purpose use. Updated lpi_live to use the extracted code.

Spent my spare moments looking over Richard's new ring buffer code for Linux native interfaces in libtrace. In particular, my aim has been test it in situations outside of the standard libtrace paradigm, e.g. using trace_event(), trace_copy_packet() and exporting over the RT protocol.

Alistair from CAIDA has updated libtrace and wdcap for capturing using the BSD native interface (something we never did, so the code was missing or half-assed). I've started integrating his changes back into both code-bases and will also look at the problem of decoding RT packets that were capturing using a native interface that is not supported by the recipient machine, e.g. BPF packets exported to a Linux host.

15

Oct

2012

Short week this week - took leave on Thursday and Friday.

Released a new version of libtrace (3.0.15) on Monday. Mostly just a few little bug and build fixes, but it had been a while since the last release. Also submitted a patch for the FreeBSD libtrace port which had been broken for a very long time.

Did a bit more refinement on my Plunge and ArimaShewhart event detectors. They're at a stage now where the number of false positives is close to none. False negatives are a bit harder to identify, of course. The next sensible step is probably to think about testing against real-time data and manually validate the events as they roll in.

Spent a day looking at the latest LPI data from a live analysis I have running on our ISP monitor. Managed to get some up-to-date stats on application usage for last September but haven't had a chance to look over it in detail yet.

I did note a bit of an increase in the amount of unknown UDP traffic, so chased up a few of the more common patterns. Have added 3 new protocols to libprotoident as a result: ZeroAccess (a trojan), VXWorks Exploit and Apple's Facetime / iMessage setup protocol.

08

Oct

2012

Libtrace 3.0.15 has been released.

This release fixes a few bugs in the previous release and adds a few minor improvements. In particular, this release fixes the problem where libtrace will claim pcap transmit is unsupported and the bug where Linux Native capture does not work on the loopback address. It also fixes some potential build errors introduced in the last release as a result of creating a separate library for libwandio.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

08

Oct

2012

Added a new anomaly detector to our network event monitor: the Plunge Detector. The basic aim is to detect situations where an otherwise active time series plunges to a very low (or zero) value. Sounds simple, but kinda tricky to do in a generic fashion. The general algorithm is track the median and minimum observed values over the past N measurements and then raise an alarm when the current value is both significantly below the median and the minimum observed values.

Spent much of the week testing both the new Plunge detector and the Shewhart detector against the various LPI time series in my test data set. Lots of refinement going on with both detectors, but starting to get pretty happy with the results.

Started working towards a new libtrace release - mostly just a few little bug fixes and tidyups. Part of the release process is to test it on a FreeBSD machine, but the old emulation image doesn't work with the new emulation network. Set up a FreeBSD 9 machine so that Brendon could make a new image, which was a lot more painful than it should have been. Managed to get libtrace tested and passed the machine over to Brendon for imaging - I expect a decent rant in his weekly report about that step of the process to :)

27

Aug

2012

Managed to get the ArimaShewhart detector fully integrated into the anomaly detection system and producing "correct" results. Now started turning my attention to using Nathan's software to provide suitable input and store measurements in a database that can be queried by the presentation / graphing side of the project.

The latest 301 assignment was due on Friday, so spent a fair bit of time helping out students who were having a few pointer difficulties.

Finished a draft revised version of my IMC paper - turns out I hadn't gone over the page limit by as much as I had feared so it was relatively easy to get the paper down to a suitable length.

Fixed a bug in libtrace relating to the use of Linux native on loopback interfaces that was reported by Asad. Might be time to think about a new release soon.

13

Apr

2012

Our paper on libtrace entitled "Libtrace: A Packet Capture and Analysis Library" has been officially published in this month's edition of ACM Computer Communication Review.

It has been a bit of a battle over the years to find a venue that was willing to publish a paper on libtrace, as the direct scientific contribution of libtrace itself is subtle. It was also difficult to articulate exactly how libtrace is so much easier and pleasant to work with compared to other trace analysis libraries. Often the improvements present in libtrace were dismissed out of hand as being nice but not necessary.

For example, capture format agnosticism was dismissed by some reviewers as mostly pointless because they never needed to work with a trace format other than pcap. The performance enhancements were similarly discredited because it was just easier to "buy a faster CPU" or because you could just use a separate zcat process to decompress the trace instead (hence the explicit discussion of the difference between using a separate process + pipe versus the threaded approach employed by libtrace).

As a result, we often had to go back to the drawing board and think more carefully about how to "sell" each of the enhancements in libtrace and clearly explain the reasoning behind each design decision. Eventually we managed to find the right combination of venue and tone that allowed us to finally get a submission accepted.
Hopefully this will lead to more network researchers learning about libtrace and adopting it for use in their own research and analysis tasks.

A copy of the paper can be downloaded from here.

13

Apr

2012

This paper introduces libtrace, an open-source software library for reading and writing network packet traces. Libtrace offers performance and usability enhancements compared to other libraries that are currently used. We describe the main features of libtrace and demonstrate how the libtrace programming API enables users to easily develop portable trace analysis tools without needing to consider the details of the capture format, file compression or intermediate protocol headers. We compare the performance of libtrace against other trace processing libraries to show that libtrace offers the best compromise between development effort and program run time. As a result, we conclude that libtrace is a valuable contribution to the passive measurement community that will aid the development of better and more reliable trace analysis and network monitoring tools.

Published in ACM Computer Communication Review, Volume 42, Issue 2 (April 2012).

Author(s): 
Shane Alcock
Perry Lorier
Richard Nelson

12

Mar

2012

Released libtrace 3.0.14 - mostly just a bug fix release. I also separated the I/O stuff into a separate library so that it can be used outside of libtrace.

Took a quick look at maji again to see if we can use it as part of the MSI project. Fixed up some bugs that became apparent when exporting lots of flow records. Also decided that maji would work a lot better if it underwent a major design change, but resisted the temptation to do so for now.

Secured the RT exporter connected to the live capture point so that only WAND machines can connect to it - someone from a lightwire address had connected to it and sent something invalid which broke the whole wdcap process. The RT exporter also now handles invalid client responses better :)

Started looking at Andreas' time series anomaly detection code. The existing system only really works with offline data, so the first goal is to get it running against a "live" input source.

07

Mar

2012

Libtrace 3.0.14 has been released.

This release fixes a few bugs in the previous release and adds a few minor improvements. Most notably, libtrace no longer assert fails when reading corrupt pcap trace files.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

20

Feb

2012

Spent most of my week working on the draft version of the paper on the effect of the CAA on DSL users. Finished the draft on Friday, having included plenty of (hopefully) interesting results. Anyone interested in reading over the paper should get in touch with me and I give you a copy.

Patched libtrace to support --with-foo configure options for all the optional dependencies. Apparently this is a bit of an issue with some Linux distros, e.g. Gentoo.

Released a new version of BSOD server on Friday to fix a crash issue that was occurring with recent libprotoident releases.

Spent some time looking at traffic that was being classed as SSL by libprotoident. Turns out that, with a bit of port and payload size analysis, I can sub-classify the SSL as Google talk, Apple push notifications, Facebook chat, PSN store, POP3S and NNTPS.

13

Feb

2012

Started working on a paper describing the results of the study I presented at NZNOG. Managed to write half of a "short" paper so far, so making reasonably good progress.

Made the necessary changes to the libtrace CCR paper and submitted a final version. One of the reviewers wanted to see more stats from the performance testing but I didn't have space to put it in. I suggested that if the editor was able to grant me more space I would include the stats.

Seemed to have a busy week supporting various software: libtrace, libprotoident etc. Glad to see plenty of people using these libraries :)

07

Feb

2012

Worked on collecting some more numbers measuring the impact of the CAA, with an eye towards writing a paper on the topic. The number of users doing P2P has also dropped dramatically, with rises in the expected categories too (such as tunneling).

Looking at the results more closely, I decided that the HTTP_P2P classification was proving to be incorrect more often than not, so traffic matching that is now treated as web rather than P2P. This change should have only a minor effect on the numbers I had presented at NZNOG.

The libtrace paper was accepted for publication in CCR. This was my fifth attempt to publish that particular paper, so pretty pleased to finally get that one done.

16

Jan

2012

Finally found and fixed the bug that was causing the occasional trace file to be truncated when written to disk. Having done that, I released libtrace 3.0.13 on Monday.

Worked with Nevil to get a test capture up and running on his capture box in Auckland. After a couple of false starts, we managed to successfully capture a day's worth of trace without issues.

Set up a Fedora machine for testing libtrace prior to subsequent releases, as it has become apparent that testing on just Debian and Ubuntu is insufficient. Will hopefully replace with a virtual machine once the new emulation network is up and running.

Started working on a possible presentation for NZNOG, mostly about libprotoident again.

Spent a little bit of time reading over my extended NAT sessions paper, making a few edits here and there.

09

Jan

2012

Libtrace 3.0.13 has been released!

This release adds support for OSPFv2, extending the libtrace API to allow easy access to OSPF headers, LSAs and Router Links and updating libpacketdump to decode OSPFv2 packets. This version also fixes some major bugs, including one where traces written using zlib were occasionally slightly truncated. A bug where trace_get_payload_from_ip was incorrectly calculating the number of bytes remaining has also been fixed.

There are also several other performance enhancements and minor bug fixes.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

09

Jan

2012

Began preparing for a new round of captures at both Auckland and our ISP. Added a feature to wdcap at Nevil's request where the amount of payload to capture can be specified in the config file (rather than being fixed at four bytes). In the process, found and fixed a libtrace bug which was causing wdcap to capture an extra four bytes of payload than what was requested.

Pushed towards a new libtrace release. First finished adding support for OSPFv2, based on Simon's code. This was a bit harder than expected, as OSPF is a rather complicated protocol and I wanted to try and get the API right first time around. There were a few little traps in the spec that Simon's original code didn't deal with very well, so I had to work around those as well. It's not a perfect implementation but seems to deal with the sample OSPF packets I have pretty well.

Started the 2012 ISP capture on Friday, seems to be going well so far.

Met with Steffen Wendzel on Friday and talked about our various projects. He was pretty impressed with libtrace and BSOD, while I expect his experience in cyber security and covert channels could be useful for us one day.

12

Dec

2011

Rather disrupted week this week, only in for three days total.

Draft version of the new sessions paper is nearly finished. Thankfully, running the old analysis against new traces has produced similar results so I can "borrow" most of the text from an old rejected paper on outbound session analysis.

Checked the results of my sleeper analysis using the longer idle time threshold. Again, not much change to the overall results but I can feel more comfortable with the distribution of idle period lengths now. Have processed the 2009 and 2011 datasets using the new threshold.

Created some anonymised versions of the ISP 2009 traces for Asad. In the process I found a weird libtrace threaded I/O bug where the last block of compressed data won't be written out before the file is closed under very specific circumstances. This one is going to be a pain to track down...

14

Nov

2011

Started collating together the results of my analysis of dark and sleeper traffic in the ISP traces. It's not finished yet, but the results I have so far can be viewed at http://www.wand.net.nz/~salcock/sleepers/

CCR rejected my libprotoident paper, primarily due to a reviewer stating that we had not compared against the "state of the art" described in a paper from 2006 (http://www-rp.lip6.fr/site_npa/site_rp/_publications/737-conextFinal.pdf). This particular technique requires no packet payload, but is only able to identify 10 different TCP application protocols (although I can supposedly create new models for other TCP applications).

I tested the default models against some ISP traffic and found that it performed much better than I had expected, but was still less accurate than the weakest of the OSS DPI techniques. Their failure rate (in terms of misclassified bytes) was 24%, compared with 4.5% for libprotoident.

Started integrating Vineyard's NAVL library into my traffic classification evaluation tool. Started out OK, but ran into a few problems with not being able to force NAVL to expire internal entries for UDP flows when I have decided the flow has ended. This creates a problem if the 5-tuple reappears later, as NAVL returns an error when I try to create a new NAVL connection for that flow because NAVL believes the flow already exists. I've filed a support request, so hopefully I'll get some sort of solution in the next day or two.

Continued integrating Simon's OSPF code into libtrace.

07

Nov

2011

Started looking into the traffic sent to "sleeper" hosts, i.e. IP addresses that have been active but are now inactive. Still putting together the initial results, but there is definitely a difference between the traffic observed heading to "dark" hosts vs the traffic observed heading to sleepers.

During the sleeper analysis, I've been able to improve a few of the libprotoident rules to correctly match more of the traffic I've been looking at.

Began integrating Simon's OSPF parsing code into libtrace. Been slightly trickier than I had anticipated due to major differences between OSPFv2 (which Simon's code parses) and OSPFv3 (which we may want to parse in future).

Had a brief phone meeting with Vineyard Networks. They've agreed to give us access to their NAVL library for evaluation.