User login

Search Projects

Project Members

Shane Alcock admin

Libtrace

Libtrace is a library for both capturing and processing packet traces. It supports a variety of common trace formats, including pcap, ERF, live DAG capture, native Linux and BSD sockets, TSH and legacy ERF formats. Libtrace also supports reading and writing using several different compression formats, including gzip, bzip2 and lzo. Libtrace uses a multi-threaded approach for decompressing and compressing trace files to improve trace processing performance on multi-core CPUs.

The libtrace API provides functions for accessing the headers in a packet directly,
up to and including the transport header.

Libtrace can also output packets using any supported output trace format, including
pcap, ERF, DAG transmit and native sockets.

Libtrace is bundled with several tools for performing common trace processing and analysis tasks. These include tracesplit, tracemerge, traceanon, tracepktdump and tracereport (amongst others).

09

Nov

2018

Libtrace 4.0.5 has been released.

This release is a collection of bug fixes and performance improvements for libtrace 4.0.4.

Major changes in this release:
* Fixed bug where only one client could subscribe to an nDAG multicast group concurrently.
* Fixed bad payload length calculations on outgoing packets when IP length field is populated by hardware.
* Fixed build error when compiling with DPDK support.
* Improved performance when decoding ETSI records via etsilive:
* Improved ERF / DAG packet processing performance by caching the framing length.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

02

Jul

2018

Libtrace 4.0.4 has been released.

New features in this release include:
* Added reference counting API for keeping track of packets shared across multiple threads.
* Added new input format for receiving live streams of packets encoded using the ETSI LI standard.
* Libpacketdump can now also decode ETSI-encoded packets.
* Added CLI option to tracereplay to "speed up" replays by a given factor, i.e. set to 2 to replay at double speed.
* Added new trace_flush_output() API function to force a libtrace output to flush any buffered output.

There are also plenty of bug fixes in this release, including:
* Fixed problems with trying to have two ring: inputs active at the same time.
* Fixed missing subseconds in timestamps from pcapng: packets.
* Fixed bug that was causing poor ndag: performance.
* Fixed bug that prevented trace_pstop() from working correctly on some live inputs.
* tracereplay will now skip meta-data records rather than trying to replay them and failing.
* Fixed some buffer overrun issues in the pcapng reading code.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

02

Mar

2018

Libtrace 4.0.3 has been released.

New features in this release include:
* A new capture format (dpdkndag:), which allows nDAG records to be intercepted and decoded using a DPDK interface.
* The message queue data structure API is now publicly exported, so can be used in user code.
* The toeplitz hasher data structure API is now publicly exported, so can be used in user code.
* Added a new API function: trace_get_perpkt_thread_id(), which will return the ID number of the running packet processing thread.
* Upgraded DAG code to use the 64 bit API, so libtrace will work with large streams.

The following bug fixes are also included:
* ERF provenance records will no longer cause libtrace to halt.
* Captures from GRE tunnel interfaces should now work correctly.
* Packets captured using DPDK will no longer lose any payload after the first 1024 bytes.
* Fixed a couple of nDAG packet corruption problems.
* Ensure all key fields are correctly initialised when doing DPDK output.
* Fixed assertion failure when libwandio has an unexpected error.

We've also further improved the performance of the nDAG format.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

18

Dec

2017

Continued trying to get better performance out of the ndag protocol. Most of my time this week was actually spent trying to resolve some issues with the DPDK packet counters, which were not proving to be particularly accurate or useful. After a lot of debugging, I realised that my DPDK packet generator software was reading and clearing the stats for all of the DPDK interfaces, not just the one it was transmitting on. Once I blacklisted the interface I was using to receive reflected ndag packets on from the packet generator, my numbers started making sense again.

Eventually I found that the best way to improve performance was to just enable jumbo frames and therefore send less multicast packets. With jumbo frames, I've been able to capture, encapsulate, export and subsequently receive a sustained 10G with no real issues.

On the OpenLI side, I've improved the API around my decoding code so that I can easily perform the main tasks that libtrace and libpacketdump will require. Specifically, I now have API functions for getting the timestamp, record length and start of the IP capture field from an ETSI-encoded packet. I've also added a "get next field as a string" function for generating nice tracepktdump output. Hopefully with all of this in place, I should be able to integrate the decoder into libtrace by the end of the year.

08

Dec

2017

Separated common code that was shared between the telescope prototype and the new wdcapsniffer into its own file so that there is less repeated code to maintain. After a couple of extra bug fixes, I've managed to get my libprotoident daily monitor code working again and now using the ndag export from wdcapsniffer as the packet source. This will help me confirm that the code is generally stable and doesn't drop packets, as I should notice fairly quickly if my libprotoident reports are empty or have bogus data in them.

Added a dpdkndag capture format to libtrace which intercepts ndag multicast on the wire using dpdk, strips the IP, UDP and ndag headers and converts the contained ERF records into libtrace packets. The idea is that this would be faster than joining the multicast group and waiting for ndag packets to work their way through the network stack. This has turned out to be the case, although it is still not enough for a client to keep up with anything more than capturing ~6.5 Gbps.

Started my OpenLI work by developing my own ETSI-LI decoder. To support this, I've written a simple DER decoder which supports most of the primitives that are present in the ETSI standards. I've also written some code to model the ETSI structural hierarchy. I can now decode an example ETSI-encoded bytestream by walking both the hierarchy and the bytestream, seeing which fields are present and interpreting them according to the type defined in the matching hierarchy entry.

08

Dec

2017

Continued polishing up my presentation on the proposed OpenLI project. The meeting itself was held on Thursday -- people seemed pretty happy with my design and thinking thus far and the project is now all scheduled to start next week. Had a good chat with Neil from the Police about the ETSI standards and some of the gotchas that I'll need to think about when writing my code.

Finished up the initial libwdcap code and used it to write a wdcapsniffer program that exports via nDAG. Spent some time testing and tweaking the wdcapsniffer on the 10g dev machines before rolling the new and improved version out onto the probe VM.

28

Nov

2017

Started working towards rolling nDAG out onto our own capture environment, so that I can observe how it performs in a slightly more realistic scenario. Had to work around a few environment limitations, such as my VM disk needing to be resized and multicast being heavily rate limited on the path between the probe and my client. I've also started working on a libwdcap that will provide all of our old snapping and anonymisation capabilities, as I won't be able to export full unencrypted packets off the probe.

Started working on an actual design and development plan for the ETSI project, including plenty of architecture diagrams. Put together some slides describing the plan and proposed architecture for presentation at the first project meeting this coming Thursday.

20

Nov

2017

Implemented a few more nDAG optimisations. I'm now using recvmmsg and sendmmsg to receive and send batches of messages using a single system call -- this saves us quite a lot of pthread cancellation status changes when doing I/O. Sender can now comfortably handle 7.5 Gbps without dropping any records. Client is still stuck around 5 Gbps receiving, but I still have one or two ideas up my sleeve.

Tested and released a new version of libtrace, which includes both nDAG read and pcap-ng read support.

Started experimenting some more with the ETSI capture format. Started adding an etsilive: input format to libtrace for testing purposes. Played with cyberprobe as a possible independent source of ETSI-encoded packets to develop my decoder against, but it turned out that their encoding was not as compliant as I had hoped. I found two errors in their encoding of the first field alone.

15

Nov

2017

Libtrace 4.0.2 has been released today.

This release adds two new input formats: pcapng and nDAG.

This release also includes the following bug fixes:
* Fixed bad IPv6 fragment offset calculation that would randomly cause IPv6 packets to be incorrectly parsed.
* Fixed bug where SIGINT would cause parallel programs to assert fail.
* Fixed compilation issues caused by inconsistent BPF presence checks.
* Fixed errors in IPv6 and SCTP libpacketdump decoders.
* Fixed bug where a short cryptopan key would cause traceanon to assert fail.
* Fixed compilation error when building traceanon against new versions of libssl.
* Fixed bug where DPDK would not be detected if it had been built as a shared library.
* Fixed bug where the wrong parallel read function would be used by libtrace.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

13

Nov

2017

Continued working on tweaking nDAG to both improve performance and add some handy features such as the ability for clients to recognise when an nDAG monitor has restarted and therefore may have missed some packets. Still got one or two ideas on how to improve performance further, so will try those out before merging the code back into mainline libtrace.

Started thinking a bit more about how my ETSI monitor is going to work and how much of it will intersect with libtrace. Will probably need to add an etsilive: read format to libtrace with suitable libpacketdump decoders to help with testing and validation, so that seems like a useful starting point.

Added a feature to my daily libprotoident analysis program to tell me what proportion of traffic on the campus network remains unidentified.

06

Nov

2017

Finished adding the core of nDAG client support to libtrace. Still a little bit of polish required before it is officially finished, but it seems to work. Managed to get around 3.5 - 4 Gbps of multicast to the libtrace client without losing anything, which is not too bad. Once I increase the data rate, it looks like the switch is dropping multicast packets rather than the client themselves so I may be starting to run into some hardware limitations.

Spent a bit of time playing around with libtasn1 and the ETSI ASN.1 specification to see how I can use the library to create some ETSI headers for packet encapsulation. Went public with a proposal for an open-source ETSI lawful intercept tool on Friday and have already got some encouraging responses.

Still seeing new patterns in the Waikato traffic, so libprotoident continues to improve. Reached 450 supported protocols this week -- next landmark is 500.

30

Oct

2017

Managed to get the new telescope software running at a decent packet rate. So far we can capture and multicast ~12 million packets per second without issues. The main limitation that prevents us from going any higher is the capacity of the 10Gb interface that we are multicasting on. Pretty happy with that result and now I can focus on ensuring that the clients will be able to keep up.

Started adding nDAG read support to libtrace. This is mostly a matter of adapting my existing test client code to work within the libtrace structure, as well as making sure that there are suitable code paths for each of the three APIs: parallel, single-threaded and event-driven.

Still seeing new protocols every week on the campus network, even with the decreasing amount of people who are present on campus. 3 new protocols this week; starting to get close to the 450 mark.

24

Oct

2017

Continued developing the new telescope software. nDAG records are now created and multicast out a specific interface. I also have a test client that is able to join the multicast groups and receive the packet streams. There's also a control channel that is used by the telescope to announce the ports that the streams will be transmitted on.

Continued tinkering with adding new libprotoident rules. Added another 6 new protocols this week, all games. Updated a few other existing rules as well to cover new variants or fix minor errors.

Had some meetings on Monday re: a possible open-source ETSI-compliant lawful intercept implementation. There's definitely some interest in the community for something open-source to exist.

17

Oct

2017

Helped Jayden with polishing up the final version of his Honours report. Hopefully he is happy with the final result!

Started testing the initial prototype of the DAG multicaster on our development boxes. Had a few issues getting dpdk pktgen to do exactly what I wanted (not helped by the terrible documentation!) but eventually managed to happily capture 10Gb of small packets split across 4 DAG streams with no real issues. Next step is to start encapsulating and multicasting some nDAG records.

Went to the STRATUS forum on Friday, flying down to Wellington on Thursday afternoon. Forum seemed to go pretty well; plenty of people that I spoke to thought that our work so far was interesting.

Released a new version of libprotoident.

09

Oct

2017

Started working on the DAG multicaster for STARDUST. Designed an encapsulation protocol for the multicaster and wrote some prototype code using the libdag API to start grabbing bunches of records and give them to the as yet unimplemented multicaster to encapsulate and send.

Spent some time reading over Jayden's honours report and gave him some (hopefully useful) feedback. The work he has done this year is really interesting; just needs a bit of literary polish so that his markers can fully appreciate it :)

Continued slowly working towards a libprotoident release. The code itself looks ready to go, so I just need to prepare the release announcements. I've updated my paper to include the extra 20 or so protocols that I've added since I started writing up the results -- the paper now covers 435 application protocols.

25

Sep

2017

Back at work after a couple of weeks disrupted by illness. Spent most of the week working on my application protocol paper. Managed to produce a few interesting looking graphs and am now starting to get a rough idea of how my narrative is going to come together. Essentially, modern application protocols are vague and therefore require a lot more work and expertise to identify. However, they are still possible to identify and there are still plenty of new protocols appearing every year, so DPI hasn't outlived its usefulness entirely yet.

Had a meeting with Alistair from CAIDA about the first steps on the STARDUST project, which is essentially a redevelopment of their telescope to support 10G capture and multiple live clients. Obviously, this is going to build a lot on our experiences so far with parallel libtrace / wdcap -- one of my key jobs will be to develop a new parallel, multicast RT protocol as the old RT protocol simply won't be fast enough anymore.

02

Jun

2017

Libflowmanager 3.0.0 has been released today.

The libflowmanager API has been re-written to be thread-safe (and therefore compatible with parallel libtrace), hence the major version number change.

The old libflowmanager API has been removed entirely; there is no backwards compatibility with previous versions of libflowmanager. If you choose to install libflowmanager 3 then you will need to update your existing code to use the new API. This should not be too onerous in most cases, as most of the old global API functions have simply been replaced with method calls to a FlowManager class instance. The README and example programs demonstrate and explain the new API in detail.

Note that much of our other software that relies on libflowmanager, such as the libprotoident tools and lpicollector, have NOT yet been officially released with libflowmanager 3 support. If you are currently using any of this software, you should continue to use libflowmanager 2.0.5 until we are able to test and release new libflowmanager 3 compatible versions.

You can download both libflowmanager 3 and libflowmanager 2.0.5 from our website.

03

Apr

2017

Have been using my new daily libprotoident email to make some good progress in terms of adding new protocols to libprotoident. Another 8 protocols added this week, with 5 existing protocols improved as well.

Found a few new bugs in my FSM tandem-repeat code after running it against my full test dataset and doing an initial validation of the resulting machines. Finished up a set of slides describing (broadly) what I'm doing overall with the FSM project and how I'm going about it, i.e. suffix trees, pattern extraction, variant detection and machine building.

Started looking into a parallel RT implementation for libtrace / wdcap, with an eye towards removing the combiner bottleneck from wdcap.

20

Mar

2017

Finished porting the remaining libprotoident tools to be parallel-compatible. Spent a couple of days looking at unknown payload patterns in some recent Uni traffic -- unfortunately I wasn't able to make much tangible progress on identifying much of the unknown traffic.

Worked on implementing an algorithm for finding tandem repeats in strings, with the eventual aim of porting it over to work with my system call sequences. The published algorithm consists of three phases, but each of those phases has either involved looking up and implementing several other string processing algorithms (LZ-decomposition, longest common extension) or has required modifications to my existing suffix tree code (extracting a suffix array, bottom-up traversal, storing the longest child suffix in each node). Therefore, I'm about half-way through implementing the algorithm.

Moved libtrace into its own github organization to reflect that libtrace is now going to be more of a community project than a WAND project. I'll still be helping out with maintaining it for now, but now the workload can be shared amongst a group of trusted libtrace users (including people outside of WAND). This will hopefully keep libtrace well looked-after, even as my available time gets more and more restricted.

06

Mar

2017

Finished testing my packet ordering fix for libtrace. Managed to come up with a more efficient method of determining an appropriate order value for int: and ring: so hopefully performance shouldn't be impacted too much by this change. Also fixed a couple of other libtrace bugs that I had noticed, particularly the horrid performance of tracertstats on some live formats. Released a new version of libtrace (4.0.1) that includes these fixes as well as a few others that have come in since the first parallel release.

More tweaks to the FSM generation code. I've found some errors in the way by which I was determining whether one machine was effectively superceded by another, which was causing me to produce extra redundant machines. I've also come up with the new method for creating the match maps when comparing two sequences -- the old method simply focused on picking the longest match and then finding any other matches that cover unmatched territory, which doesn't work so well for some looping sequences which have repetitive sub-sequences within them. My new method tries to find an optimal set of matches that gets the best possible coverage while minimising the amount of overlap, so we avoid matches that only end up covering one token because the rest of the sequence has already been covered by a larger match.