User login

Shane Alcock's Blog

15

Nov

2017

Libtrace 4.0.2 has been released today.

This release adds two new input formats: pcapng and nDAG.

This release also includes the following bug fixes:
* Fixed bad IPv6 fragment offset calculation that would randomly cause IPv6 packets to be incorrectly parsed.
* Fixed bug where SIGINT would cause parallel programs to assert fail.
* Fixed compilation issues caused by inconsistent BPF presence checks.
* Fixed errors in IPv6 and SCTP libpacketdump decoders.
* Fixed bug where a short cryptopan key would cause traceanon to assert fail.
* Fixed compilation error when building traceanon against new versions of libssl.
* Fixed bug where DPDK would not be detected if it had been built as a shared library.
* Fixed bug where the wrong parallel read function would be used by libtrace.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

13

Nov

2017

Continued working on tweaking nDAG to both improve performance and add some handy features such as the ability for clients to recognise when an nDAG monitor has restarted and therefore may have missed some packets. Still got one or two ideas on how to improve performance further, so will try those out before merging the code back into mainline libtrace.

Started thinking a bit more about how my ETSI monitor is going to work and how much of it will intersect with libtrace. Will probably need to add an etsilive: read format to libtrace with suitable libpacketdump decoders to help with testing and validation, so that seems like a useful starting point.

Added a feature to my daily libprotoident analysis program to tell me what proportion of traffic on the campus network remains unidentified.

06

Nov

2017

Finished adding the core of nDAG client support to libtrace. Still a little bit of polish required before it is officially finished, but it seems to work. Managed to get around 3.5 - 4 Gbps of multicast to the libtrace client without losing anything, which is not too bad. Once I increase the data rate, it looks like the switch is dropping multicast packets rather than the client themselves so I may be starting to run into some hardware limitations.

Spent a bit of time playing around with libtasn1 and the ETSI ASN.1 specification to see how I can use the library to create some ETSI headers for packet encapsulation. Went public with a proposal for an open-source ETSI lawful intercept tool on Friday and have already got some encouraging responses.

Still seeing new patterns in the Waikato traffic, so libprotoident continues to improve. Reached 450 supported protocols this week -- next landmark is 500.

30

Oct

2017

Managed to get the new telescope software running at a decent packet rate. So far we can capture and multicast ~12 million packets per second without issues. The main limitation that prevents us from going any higher is the capacity of the 10Gb interface that we are multicasting on. Pretty happy with that result and now I can focus on ensuring that the clients will be able to keep up.

Started adding nDAG read support to libtrace. This is mostly a matter of adapting my existing test client code to work within the libtrace structure, as well as making sure that there are suitable code paths for each of the three APIs: parallel, single-threaded and event-driven.

Still seeing new protocols every week on the campus network, even with the decreasing amount of people who are present on campus. 3 new protocols this week; starting to get close to the 450 mark.

24

Oct

2017

Continued developing the new telescope software. nDAG records are now created and multicast out a specific interface. I also have a test client that is able to join the multicast groups and receive the packet streams. There's also a control channel that is used by the telescope to announce the ports that the streams will be transmitted on.

Continued tinkering with adding new libprotoident rules. Added another 6 new protocols this week, all games. Updated a few other existing rules as well to cover new variants or fix minor errors.

Had some meetings on Monday re: a possible open-source ETSI-compliant lawful intercept implementation. There's definitely some interest in the community for something open-source to exist.

17

Oct

2017

Helped Jayden with polishing up the final version of his Honours report. Hopefully he is happy with the final result!

Started testing the initial prototype of the DAG multicaster on our development boxes. Had a few issues getting dpdk pktgen to do exactly what I wanted (not helped by the terrible documentation!) but eventually managed to happily capture 10Gb of small packets split across 4 DAG streams with no real issues. Next step is to start encapsulating and multicasting some nDAG records.

Went to the STRATUS forum on Friday, flying down to Wellington on Thursday afternoon. Forum seemed to go pretty well; plenty of people that I spoke to thought that our work so far was interesting.

Released a new version of libprotoident.

09

Oct

2017

Libprotoident 2.0.12 has been released.

This release is mostly a protocol database update. We've added 26 new protocols and updated a further 33 others since the last release.

We've also added a new category for IP Camera protocols. Some already existing protocols have been moved into this category to better reflect their purpose.

The full list of updated protocols can be found in the libprotoident ChangeLog.

Download libprotoident 2.0.12 here!

09

Oct

2017

Started working on the DAG multicaster for STARDUST. Designed an encapsulation protocol for the multicaster and wrote some prototype code using the libdag API to start grabbing bunches of records and give them to the as yet unimplemented multicaster to encapsulate and send.

Spent some time reading over Jayden's honours report and gave him some (hopefully useful) feedback. The work he has done this year is really interesting; just needs a bit of literary polish so that his markers can fully appreciate it :)

Continued slowly working towards a libprotoident release. The code itself looks ready to go, so I just need to prepare the release announcements. I've updated my paper to include the extra 20 or so protocols that I've added since I started writing up the results -- the paper now covers 435 application protocols.

03

Oct

2017

Spent a decent portion of my week working on my reworked cluster evaluation code for STRATUS. The new version seems to be producing labels that are much more useful, so my ability to evaluate clusters and identify the least conforming members has improved greatly.

Continued to tweak and improve the libprotoident rules. Started working towards a possible 2.0.12 release by updating documentation and running some basic build tests on various operating systems.

25

Sep

2017

Back at work after a couple of weeks disrupted by illness. Spent most of the week working on my application protocol paper. Managed to produce a few interesting looking graphs and am now starting to get a rough idea of how my narrative is going to come together. Essentially, modern application protocols are vague and therefore require a lot more work and expertise to identify. However, they are still possible to identify and there are still plenty of new protocols appearing every year, so DPI hasn't outlived its usefulness entirely yet.

Had a meeting with Alistair from CAIDA about the first steps on the STARDUST project, which is essentially a redevelopment of their telescope to support 10G capture and multiple live clients. Obviously, this is going to build a lot on our experiences so far with parallel libtrace / wdcap -- one of my key jobs will be to develop a new parallel, multicast RT protocol as the old RT protocol simply won't be fast enough anymore.