User login

Blogs

16

May

2017

Added options to the ampweb mesh configuration to allow setting the individual tests that should be visible in the matrix view, and whether the mesh is a source for these tests or not. Previously we tried to guess and enable these as tests were scheduled, but this led to issues when meshes were created for convenient display grouping and didn't actually run any tests, and it was not transparent why things were behaving the way they did. This is now an entirely manual process that the user has full control over. Also fixed a bug in the matrix display that meant the throughput matrix was losing configuration options when switching between it and other tests, and updated the URL validity checks to match the new formats so that our URLs are now pushed into browser history.

Spent some more time investigating why the BGP code was so slow. Most of the time appears to be spent copying route entries, so I rewrote the deepcopy function for that class to be much more efficient, and also reduced the number of locations where route entries were copied. Replaced some dictionaries with defaultdict structures that remove the need to check for key presence (in very large data structures) before taking actions. I can now import 1 million routes from a peer in under 60 seconds, including running them through a number of simple filters and copying them into a number of VRFs. Exporting these routes takes more memory than I have available however, which will be a job for next week.

16

May

2017

Added interface elements to ampweb enabling the scheduling of a normal or HTTP POST style throughput test, as well as the database/ampy support to make this work. Updated the graph browser to allow selecting and displaying throughput tests of both sorts as well. Spent some time trying to add support for the new tests to the matrix as well, but we haven't done this particular sort of data split before and it's not immediately clear the best way to go about this.

Started to look into why my BGP code is scaling so poorly at 10,000+ prefixes.

08

May

2017

Added another 5 protocols to libprotoident -- having a slightly more powerful PC for installing and running various candidate applications has helped quite a bit. Updated the rules for several more protocols as well.

Made some more progress on my protocol taxonomy -- I'm up to 'P' for the TCP protocols so I'm probably about 1/4 of the way through now.

Continued re-factoring the FSM generation code. Getting close to done, although I suspect the amount of changes and variable renaming will require a fair bit of testing to make sure I've transferred everything across correctly.

Added the ability to choose between TCP and HTTP throughput data on the AMP matrix. To do this, I had to bring the amp-web/nntsc install on prophet back up to date after a few months of being untouched. As always, there were a few issues with dependencies and versioning which slowed everything down, but eventually Brendon and I got it all working correctly.

01

May

2017

Another disrupted week, this time due to being ill. Spent most of my available time looking over the output of my new multi-process state machine generation algorithm. The extra sequence fragments that become apparent when considering multiple processes managed to reveal a few new situations where my code wasn't quite doing the right thing. I've fixed those and am reasonably happy again with the machines produced for my test dataset.

Moved on to some code re-factoring, as the existing code-base had become a bit of a mess from hacking in fixes to all of the edge cases I had been dealing with. In particular, I'm aiming to separate code that deals with the machine itself, i.e. the states and their transitions, from the code that compares sequences and determines what needs to be added to (or removed from) the machine to accommodate the variation.

24

Apr

2017

Improved route aggregation to include the AS set of all ASNs involved in the aggregation so that peers can better perform loop detection.

Improved community support so that imported communities are now in a useful format, and can also now be exported to peers. Added a new filter to match the commonly used no-export communities.

Improved handling of withdraw messages to deal with supernets of advertised prefixes being specified - we can't just remove the exact prefix sent by the peer. Also tidied up some other prefix matching that was using the overlaps() function rather than being a strict subset.

Tidied up AS path modification and community modification via filters to make copies of the route entries so that the changes are only temporary and not reused between different peers and VRFs. A clean original copy of the route is kept and modifications are applied to that rather than stacking up repeatedly on the same instance.

24

Apr

2017

Tidied up exporting routes to peers to remove some that should not be sent - routes should not be advertised back to the peer we got them from in the first place. Also started to filter routes by VRF as well, so that peers can be limited to which they receive routes from.

Started to build up a fake network topology based on the REANNZ network, with different peers and different relationships between them in order to make sure that the required capabilities are present to build a realistic network.

24

Apr

2017

Spent the week working on improving the ability to filter and modify routes as they arrive from a peer, are imported or exported to a VRF, or exported to other peers. Peers and VRFs now have a list of filters that are applied to incoming/outgoing routes. The filters contain lists of rules which are processed in order until one matches (based on prefix, peer, origin, etc), at which point the actions are applied to the rule to add/remove communities or to modify the AS path, and the route is accepted or rejected. If a filter fails to match then the next filter in the list is applied until eventually one matches or they all fail and the default behaviour occurs. Exported routes can now also be aggregated.

19

Apr

2017

Slightly disrupted week with Easter and cyclones having an impact on the productivity. Most of my time ended up being spent hunting down more previously unknown protocols. Just three new protocols this week, along with fixes for three more.

On the STRATUS side, I worked on creating a way to "combine" the suffix trees for each individual process so that we can account for sequences that appear frequently in the whole dataset but never more than once or twice within a given process. The original implementation would not recognise those sequences as frequent, because it considered each process individually. I think I've got this working now -- but I'm yet to look at the results too closely.

10

Apr

2017

Continued delving into the unknown traffic on the campus network. Had a mix of frustrating days and successful days -- one protocol (N2Ping) took nearly two days to track down but I got there in the end. 8 new protocols added to libprotoident this week again, so we're starting to get close to 400 supported protocols in libprotoident.

Another week of refinement on the FSM code. Most of the effort has been focused on loop recognition, particularly in terms of making sure we don't ignore candidates that can be used to identify loops.

06

Apr

2017

Put the finishing touches to the AMP throughput test before building some new packages for testing. Updated the collector packages to understand the new options so that the data can be accessed/saved.

Spent some time trying to integrate VRFs into the BGP project. Routes are currently placed into a number of VRFs based on the peer, and the VRFs that routes are exported from are selected based on the peer ASN and address.

Figured out an event in Chrome that I can easily hook into to determine when a Youtube video has ended (rather than polling and hoping), and wrote code to do so. I now have a standalone program that will tell me how much time was spent in initial buffering, later stalling/rebuffering, playback, etc. Started adding a timeline of actions to show how much time was spent in each state, and when. After that will likely be the task of trying to get this running as part of AMP (which could be annoying, at least until headless mode is available in packaged versions).