User login

Blogs

09

Nov

2017

Found and fixed a bug where the udpstream test would not receive all the reflected packets used for RTT calculations because it gave up listening after waiting after one inter-packet gap. It now waits until all the packets have been received or the global loss timeout is reached (multiple seconds). Also found and fixed the problem that led to discovering this - the web interface was asking for inter-packet gap in milliseconds and then treating the number entered as if it was in microseconds, leading to a gap 1000 times smaller than expected.

Added the ability for an ampweb user to change their own details through the web interface without requiring admin privileges. Spent some time testing that the new permissions model works correctly and that users are limited appropriately.

Started work on adding debconf support to the ampy package as a simple way to ensure there is a usable user right from the start without hardcoding one.

07

Nov

2017

Last week I kept on reading through more traffic engineering papers. The papers that I am currently looking through on TE cover different network types as well as look at various TE goals, such as resource utilisation optimisation, QoE maximisation and congestion minimisation. They were found from a literature review paper that looks at how SDN can benefit TE. The papers that I am focusing on look at OpenFlow and TE. Last week I have also finished and ran my Fast-Failover group timing tests.

This week I am planning on finishing off reading the TE with OpenFlow papers from the literature review. I also want to try and run more tests and see if I can get a sense of how long a recovery based error detection method (not using fast failover groups and precomputation) takes to complete. I currently have results for protection and would like to compare them with recovery. I am also currently in the process of looking for source code for some of the error detection and recovery systems presented in the papers I have read through. I would like to run some of these systems or methods and potentially assess their behaviour, problems and performance.

06

Nov

2017

Finished adding the core of nDAG client support to libtrace. Still a little bit of polish required before it is officially finished, but it seems to work. Managed to get around 3.5 - 4 Gbps of multicast to the libtrace client without losing anything, which is not too bad. Once I increase the data rate, it looks like the switch is dropping multicast packets rather than the client themselves so I may be starting to run into some hardware limitations.

Spent a bit of time playing around with libtasn1 and the ETSI ASN.1 specification to see how I can use the library to create some ETSI headers for packet encapsulation. Went public with a proposal for an open-source ETSI lawful intercept tool on Friday and have already got some encouraging responses.

Still seeing new patterns in the Waikato traffic, so libprotoident continues to improve. Reached 450 supported protocols this week -- next landmark is 500.

31

Oct

2017

Last week I carried on reading through a few more traffic engineering papers and have also looked at a couple of SDN controller performance evaluation papers. At the end of the week, I started to set-up VM environments to run some quick tests and hopefully benchmark some of the solutions available for error detection.

This week I have a few more TE papers to go through. I also want to run some tests to find out how long the fast failover group takes to switch to a new bucket when a link goes down. I also want to try and run a few benchmarks on some the current error detection and recovery methods/systems from the papers I have read.

30

Oct

2017

Managed to get the new telescope software running at a decent packet rate. So far we can capture and multicast ~12 million packets per second without issues. The main limitation that prevents us from going any higher is the capacity of the 10Gb interface that we are multicasting on. Pretty happy with that result and now I can focus on ensuring that the clients will be able to keep up.

Started adding nDAG read support to libtrace. This is mostly a matter of adapting my existing test client code to work within the libtrace structure, as well as making sure that there are suitable code paths for each of the three APIs: parallel, single-threaded and event-driven.

Still seeing new protocols every week on the campus network, even with the decreasing amount of people who are present on campus. 3 new protocols this week; starting to get close to the 450 mark.

24

Oct

2017

Updated the AMP user UI to allow users to view/modify their own details, which required changing the way permissions were tested in a few places to properly control access. Tidied up some of the modal dialogs to properly update the different parts of the form in response to user input, hopefully making it easy to see what needs to be fixed/completed before the form can be submitted.

Spent some time tidying up the source for the BGP router and trying to make sure that the style is consistent across all the source files. Also ran some static analysis/lint tools over the source to help make sure we are being sensible.

24

Oct

2017

Expanded the new AMP user management interface to allow different roles to be given to each user, splitting the ability to view configuration from being able to edit it. Added and tested all the backend parts required to make the user management work - add/remove/modify/etc users. Updated each of the front end components to expect the correct level of permissions.

Continued to work on the BGP design document taking some feedback into account.

Started organising the BGP code into a more sensible looking module with tests and code in standard locations. Spent a lot of time getting the test setup working properly when run from setup.py. It appears that the default setuptools test loader wants to treat every single file as a test rather than just those that match the documented filter, so this had to be changed to exclude non-test files. This was also complicated by the fact that the majority of the code is python3 only but some tests need supporting elements that only run in python2, and so parts need to be skipped based on the version of python being used.

24

Oct

2017

Spent most of this week working on generating email events from AMP data. Moved the user filters (configurable via the website) into the eventing database so that they can be loaded by the eventing processes and filter events in realtime. The aim is to have email alerts triggered by sensible filters in the backend rather than triggering on any group that crosses a threshold in size. The web front end does a lot of work trying to improve the quality of the event groups that I've not replicated, but host filtering and event types will now be taken into account.

Added a basic web UI to configure users so that they can have email addresses associated with them for alerting.

24

Oct

2017

Started looking into prometheus as an option for extracting useful statistics out of the BGP router. It's pretty simple to get working, though doing monitoring across multiple processes isn't as clean as I would like. Added tracking of simple route/prefix statistics to a test branch, and had a think about ways to get more detailed routing information (such as a looking glass might present) without interfering with updating routes.

Updated the pregenerated/prepackaged martians filter to include IPv6 martians.

Investigated an issue with some AMP graphs that prevents the interactive sliders from working on certain browser/OS combinations (mostly Windows). Hard to replicate. Looked like it might be an issue with out of date javascript libraries, but updating them didn't fix it. Need to find an easier way to replicate it to get any further I expect.

Spent a bit more time working on the design document from last week.

24

Oct

2017

Continued developing the new telescope software. nDAG records are now created and multicast out a specific interface. I also have a test client that is able to join the multicast groups and receive the packet streams. There's also a control channel that is used by the telescope to announce the ports that the streams will be transmitted on.

Continued tinkering with adding new libprotoident rules. Added another 6 new protocols this week, all games. Updated a few other existing rules as well to cover new variants or fix minor errors.

Had some meetings on Monday re: a possible open-source ETSI-compliant lawful intercept implementation. There's definitely some interest in the community for something open-source to exist.