User login

Shane Alcock's Blog

15

Aug

2011

Determined that the reason libpcap was outperforming libtrace when running the scan analysis was because we were CPU-bound rather than IO-bound. This meant that the faster IO of libtrace was not providing enough gain to cancel out the overhead of the libtrace function calls (compared with the direct pointer manipulation I was doing in my pcap program).

As a result, I decided to also test the libraries by doing a simple packet/byte count for each TCP and UDP port which would turn out to be IO-bound instead. In this case, libtrace was much faster than libpcap. Also implemented the two analyses using libcoral and ruby-libtrace. Libcoral was both slower and required more LOC than libtrace for both tests. Ruby-libtrace required less code for the port count (but more for the scan study, as I needed to write bindings for the flow management library I was using) but was waaaayyyy slower to run.

Finally finished running the longitudinal analysis on the various ISP traces and started working on adding the resulting graphs to my webpages. Decided that the ISP C time series graphs would be best done by plotting each year separately with an X-axis defined by the date minus the year, e.g. http://www.wand.net.nz/~salcock/longitude/graphs/icmp/icmp_in_ispc.png . This, of course, involved reworking a decent chunk of my graph generation scripts...

08

Aug

2011

Short week this week, as I was in Wellington on Thursday and Friday.

Managed to get Bro running and producing results that I could replicate with a libtrace program. Found that Bro was tracking TCP state incorrectly - it would often describe a TCP flow as both established and closed correctly when, in fact, no SYNs were observed at all. Reported the bug to the Bro team and decided to use my state classifications from now on.

Wrote a libpcap program that was equivalent to the libtrace program to compare the performance of the two. Surprisingly, the "zcat | dagconvert | libpcap" run was quite a bit faster than the "libtrace" equivalent. Profiled the libtrace program and managed to find a couple of opportunities for speeding things up, mostly through increased caching. The libpcap program is still slightly faster now, but the gap has closed significantly.

01

Aug

2011

Started getting some results from processing various Auckland and ISP traces - found one or two bugs along the way, so some re-processing has been necessary again.

Finished automating the graphing part of the analysis.

Continued working on an AS-level analysis for the trace data. Reading the routeviews BGP data is still not going well - it works in the general case but sooner or later you end up hitting a record or update that doesn't make sense and the whole thing segfaults.

Received reviews for the rejected libtrace paper. In response, I've started looking into replicating the simple Allman / Paxson study that originally used Bro for extract the required packet and flow properties. The current plan is replicate the study using each of the packet processing libraries mentioned by reviewers as equivalent to libtrace and prove once and for all that those libraries are nowhere near as good as libtrace.

25

Jul

2011

Released maji-1.0.2 on Monday.

Continued processing trace sets to generate results for my graphs. I soon realised that manually updating the scripts that convert the results into graphs was taking a horrendously long time (and was also incredibly tedious), so started working on a different script that could automatically generate the bash and gnuplot scripts I had been using. I can then just add a single line to the script for each new dataset and the graphs will be created from there.

This hasn't been entirely straightforward - each different type of graph has subtle differences that prevent me from using a single generic "make CDF" or "make time series" function. Still, it seems to be working well, just need to add code for the last few analysis types that I've done.

Found out over the weekend that the libtrace paper was rejected by IMC again, but the inbound sessions paper was accepted by ATNAC.

18

Jul

2011

After a two year development hiatus, a new version of maji (our libtrace-based IPFIX meter) has finally been released.

This release adds support for encrypting IPFIX records exported using the TCP and UDP transports, fixes some bugs observed when measuring IPv6 traffic, adds new information elements for ICMPv6 and also fixes a few errors and warnings that have come about due to changes in supporting software over the past couple of years.

We would like to offer special thanks to Benjamin Black and Rong Zheng, who both contributed code towards this release of maji.

The full list of changes is described in the maji ChangeLog.

Download maji-1.0.2 here!

18

Jul

2011

Continued processing Waikato traces and adding more graphs to http://www.wand.net.nz/~salcock/longitude/ . Had a few segfault-related hiccups along the way which has slowed things down quite a bit.

Worked on a new analysis test that would divide traffic by source and destination AS using MIBs downloaded from routeviews. Turned out that libwandbgp couldn't really support maintaining a BGP route table from routeviews data for any length of time, so had to re-write a decent chunk of the library to be able to do so. Still not quite working yet, but starting to get there.

Received some patches for both libtrace and maji from one of our users, which I checked and integrated back into our software for future release.

11

Jul

2011

Continued to have a few problems with processing large trace sets. Finally managed to get rid of all the memory leaks in my analysis code and fixed a tricky little libtrace bug that would cause processing to stop as soon as it hit a trace file less than 1 MB in size.

Started working on a web page to display all the graphs I'm creating nicely - http://www.wand.net.nz/~salcock/longitude/ . At the moment, only some of the Waikato I results are up, but will continue adding more results over the course of the week.

Tested and released new versions of both libtrace and libprotoident.

Started updating libwandbgp to be able to read bzip2 compressed files (such as the ones released by routeviews).

06

Jul

2011

It's a busy day of software releases - libprotoident 2.0.2 has also been released!

This release further improves the range of protocols matched by libprotoident, as well as improving the rules for some existing ones. There is also a new tool included with libprotoident, lpi_live, that classifies flows as soon as possible (rather than waiting for the flow to expire, as lpi_protoident does) and thus is more useful for real-time analysis.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.2 here!

06

Jul

2011

Libtrace 3.0.11 has been released!

This release adds support for ECN bits in the TCP header, fixes a notorious bug where trace format auto-detection failed on small trace files and fixes several problems with BPF filters and the event API.

Note: due to the changes in the TCP header, some libtrace programs that examine the reserved bits in the TCP header may not build against libtrace 3.0.11 (especially any code that did so because we didn't support ECN previously!). We apologise for any inconvenience resulting from this change.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

04

Jul

2011

Had a few problems processing Waikato I with all my analysis modules, mainly due to memory consumption. Did manage to get results for some of the analyses though and
worked on developing scripts to turn those results into useful graphs.

I blogged about some of the more interesting graphs here: http://www.wand.net.nz/content/messing-around-waikato-i

Found and fixed a libtrace bug where the automatic format detection would fail for small trace files - this one was bugging me for a while and cropped up again during the Waikato I analysis. Thinking about doing another release in the near future.