User login

Blogs

15

Dec

2017

Found and fixed a couple of small bugs in the pickling implementation of the RouteEntry class used in the BGP router. Updated the unit tests to check that prefixes and route entries could be correctly pickled and unpickled. Also found and fixed various small issues that didn't show up in testing, but did when exposed to real BGP implementations and a more diverse set of routes (more tests required!).

Started work on getting useful performance numbers around how long it takes to process and distribute routes, so merged the testing prometheus code I had previously written and expanded it to cover more of the interesting parts of the code. Every time routes are touched (importing, exporting, filtering, etc) the time that took is recorded and available to query. So far it looks like most of the time spent is outside of my main functions and in other places - moving data around between processes.

15

Dec

2017

Had another look into building an AMP test using headless Chrome to measure web (particularly YouTube) performance. I can get my code building within the Chrome build system, but I really want to create a library that I can link my own code against, and nothing like that gets built. They claim it does, but those libraries are missing most of the symbols I need, so still need to look into this further.

Found and fixed an issue around amplet2-client cert fetching failing after a certain number had been issued. Turned out to be a simple type issue and comparisons were being made using the wrong type, thus sorting incorrectly and returning an incorrect certificate.

Spent some time writing installation documentation for the AMP server components and adding it to the github wiki.

15

Dec

2017

Built and released new Debian and Ubuntu packages for amplet2-client, ampy, and ampweb.

Found and fixed a few issues in the netevmon email filtering that were caused by the incorrect types being used to make comparisons. Built and installed new packages in one deployment for testing.

Started work tidying up the C modules recently created for some of the more memory hungry parts of the BGP router. Was able to simplify it in a few places, reorganising code to be able to replace custom code with existing library functions, and shrink the amount of memory required slightly further again.

08

Dec

2017

Separated common code that was shared between the telescope prototype and the new wdcapsniffer into its own file so that there is less repeated code to maintain. After a couple of extra bug fixes, I've managed to get my libprotoident daily monitor code working again and now using the ndag export from wdcapsniffer as the packet source. This will help me confirm that the code is generally stable and doesn't drop packets, as I should notice fairly quickly if my libprotoident reports are empty or have bogus data in them.

Added a dpdkndag capture format to libtrace which intercepts ndag multicast on the wire using dpdk, strips the IP, UDP and ndag headers and converts the contained ERF records into libtrace packets. The idea is that this would be faster than joining the multicast group and waiting for ndag packets to work their way through the network stack. This has turned out to be the case, although it is still not enough for a client to keep up with anything more than capturing ~6.5 Gbps.

Started my OpenLI work by developing my own ETSI-LI decoder. To support this, I've written a simple DER decoder which supports most of the primitives that are present in the ETSI standards. I've also written some code to model the ETSI structural hierarchy. I can now decode an example ETSI-encoded bytestream by walking both the hierarchy and the bytestream, seeing which fields are present and interpreting them according to the type defined in the matching hierarchy entry.

08

Dec

2017

Continued polishing up my presentation on the proposed OpenLI project. The meeting itself was held on Thursday -- people seemed pretty happy with my design and thinking thus far and the project is now all scheduled to start next week. Had a good chat with Neil from the Police about the ETSI standards and some of the gotchas that I'll need to think about when writing my code.

Finished up the initial libwdcap code and used it to write a wdcapsniffer program that exports via nDAG. Spent some time testing and tweaking the wdcapsniffer on the 10g dev machines before rolling the new and improved version out onto the probe VM.

28

Nov

2017

Started working towards rolling nDAG out onto our own capture environment, so that I can observe how it performs in a slightly more realistic scenario. Had to work around a few environment limitations, such as my VM disk needing to be resized and multicast being heavily rate limited on the path between the probe and my client. I've also started working on a libwdcap that will provide all of our old snapping and anonymisation capabilities, as I won't be able to export full unencrypted packets off the probe.

Started working on an actual design and development plan for the ETSI project, including plenty of architecture diagrams. Put together some slides describing the plan and proposed architecture for presentation at the first project meeting this coming Thursday.

20

Nov

2017

Implemented a few more nDAG optimisations. I'm now using recvmmsg and sendmmsg to receive and send batches of messages using a single system call -- this saves us quite a lot of pthread cancellation status changes when doing I/O. Sender can now comfortably handle 7.5 Gbps without dropping any records. Client is still stuck around 5 Gbps receiving, but I still have one or two ideas up my sleeve.

Tested and released a new version of libtrace, which includes both nDAG read and pcap-ng read support.

Started experimenting some more with the ETSI capture format. Started adding an etsilive: input format to libtrace for testing purposes. Played with cyberprobe as a possible independent source of ETSI-encoded packets to develop my decoder against, but it turned out that their encoding was not as compliant as I had hoped. I found two errors in their encoding of the first field alone.

15

Nov

2017

Libtrace 4.0.2 has been released today.

This release adds two new input formats: pcapng and nDAG.

This release also includes the following bug fixes:
* Fixed bad IPv6 fragment offset calculation that would randomly cause IPv6 packets to be incorrectly parsed.
* Fixed bug where SIGINT would cause parallel programs to assert fail.
* Fixed compilation issues caused by inconsistent BPF presence checks.
* Fixed errors in IPv6 and SCTP libpacketdump decoders.
* Fixed bug where a short cryptopan key would cause traceanon to assert fail.
* Fixed compilation error when building traceanon against new versions of libssl.
* Fixed bug where DPDK would not be detected if it had been built as a shared library.
* Fixed bug where the wrong parallel read function would be used by libtrace.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

13

Nov

2017

Continued working on tweaking nDAG to both improve performance and add some handy features such as the ability for clients to recognise when an nDAG monitor has restarted and therefore may have missed some packets. Still got one or two ideas on how to improve performance further, so will try those out before merging the code back into mainline libtrace.

Started thinking a bit more about how my ETSI monitor is going to work and how much of it will intersect with libtrace. Will probably need to add an etsilive: read format to libtrace with suitable libpacketdump decoders to help with testing and validation, so that seems like a useful starting point.

Added a feature to my daily libprotoident analysis program to tell me what proportion of traffic on the campus network remains unidentified.

09

Nov

2017

Spent some time working on the packaging scripts to upgrade ampy/ampweb/netevmon to the newest version, including moving some database tables around, populating new tables and dealing with the debconf answers during install. Installed these multiple times on a few different Debian flavours while trying to make sure that they all work.

Found and fixed a few issues in the ampweb matrix that were preventing the udpstream data from displaying properly. There are still a few issues here around udpstream data being used to generate latency graphs, but it's getting closer to working in every case.

Continued to work on tidying up some of the BGP router code.